LLM Evaluation Metrics

LLM evaluation metrics are numbers you can defend on your tasks. They are not a reprint of an academic leaderboard, and they are not “the model scored 8.4 for helpfulness.”

This page is the measurement layer under LLM evaluation services. If you came here for LLM as a judge, read that next — a judge is one scorer, not the metric catalogue.

Start from the failure, not the acronym

Pick the metric that would have caught the last incident:

• The API downstream could not parse the reply → schema / format pass rate
• The assistant invented a policy → groundedness against a source, or hallucination testing
• It answered a question it should have refused → refusal / out-of-scope rate
• A prompt tweak made Tuesday worse than Monday → version delta on a frozen set

If you cannot name the failure, you are not ready for a dashboard. Write the rubric first.

Metrics that usually earn their keep

Task pass rate

A binary or graded score against a written definition of done for that scenario. “Did it book the right slot / extract the right fields / follow the system prompt.” This is the metric product people can argue about.

Format and contract checks

JSON parses. Required keys present. Length bounds. Citation markers exist when you asked for them. These should be code, not a judge.

Groundedness / support

Where a source of truth exists (retrieved chunks, a knowledge article, a database row): does the answer stay inside it? This is closer to RAG evaluation than to a vibe score.

Refusal and over-refusal

Two numbers. Did it refuse the prompts it should refuse? Did it refuse the prompts a user should get an answer to? One without the other is how you ship a mute assistant or a reckless one.

Inter-rater agreement

If two humans (or a human and a judge) cannot agree on the rubric, the metric is theatre. Measure agreement on a slice before you scale.

Version delta

Same set, new prompt, model, or temperature. Report what moved and a few examples. A single “quality score” that hides regressions is how teams ship a worse model with a better slide.

Metrics that usually do not

• BLEU / ROUGE as the product KPI. Useful in some research settings. Rarely what a user meant.
• Public MMLU-style scores as proof your checkout assistant works.
• A 1–5 “helpfulness” with no anchors and no examples.
• Latency and cost as quality. Track them. Do not confuse them with correctness.

We do not publish a Nexus Eclipse metric pack with invented weights. Your weights follow your risk.

How to report without lying

A usable eval report names:

Field	Why
Set ID and date	So nobody “updates the examples” mid-argument
n	A score on twelve rows is an anecdote
Split	Happy path vs edge vs adversarial
Method	Code / human / judge
Residual risk	What the metric cannot see

Parent work: AI testing. If the product also calls tools, add agent testing — those failures will not show up in a text-only pass rate.

FAQs

What is the one LLM evaluation metric I should start with?

Task pass rate on a frozen set of real product prompts, plus format checks if you have a contract. Add groundedness when answers must stay on a source.

Are LLM-as-a-judge scores a metric?

They are a source of scores. Treat them like a junior reviewer: useful after calibration, dangerous as the only number.

Do you sell a metrics platform?

No. We run evaluation on your product and leave you the set and the rubric. Tooling is whatever you already use.

How do we start?

Book a QA consultation. Bring the last wrong answer that shipped and the prompt that produced it. We will name two metrics that would have caught it — or tell you the problem is not measurement yet.