LLM as a Judge

LLM-as-a-judge means you ask a language model to score, rank, or pass/fail another model’s output against a written rubric. It is a scoring method. It is not a product, not a certification, and not a substitute for knowing what “correct” means in your workflow.

We use it on LLM evaluation engagements when the rule is too fuzzy for a regex and too expensive to send every row to a human. We do not treat the judge as ground truth.

What people actually want

Teams search “LLM as a judge” after one of three moments:

• A prompt change “felt better” and nobody can prove it
• Human review of a 400-row eval set will miss the release
• A vendor demoed an auto-score and called it evaluation

The useful question is not “which judge model is SOTA.” It is: what must be true for this answer to ship, and can a second model reliably see that?

When a judge is a reasonable tool

A judge can help when:

• The rubric is written in the same language as the output (tone, structure, “did it follow the system prompt”)
• You can spot-check a sample of judge decisions against a human
• You are comparing two versions on the same frozen set, not inventing a public ranking
• Failure is cheap enough that a wrong score is an annoyance, not a clinical or payments incident

Pairwise “which answer is better for this rubric” is often more stable than a 1–5 score with no anchors.

When the judge is the wrong tool

Do not use LLM-as-a-judge as the only check when:

• Factuality is the risk and you have a source of truth (docs, a database, a retrieved chunk). Score against the source. See hallucination testing and RAG evaluation.
• Format is mechanical. JSON that must parse, a required field, a max length — that is code, not a judge.
• Safety bypasses are the job. A polite judge will miss jailbreaks. That is AI red teaming.
• The judge is the same family as the candidate and you have not measured whether they fail together.

A judge that shares the candidate’s blindness will give you a green dashboard and a wrong product.

How to run it without lying to yourself

1. Write the rubric first. Pass/fail criteria a human would use. “Helpful” is not a criterion.
2. Freeze the set. Real product tasks, plus the ugly ones. Version it.
3. Mechanical checks first. Schema, citations present, refusal on out-of-scope. See LLM evaluation metrics.
4. Judge the residue. Only the rows that need judgment.
5. Calibrate. A human scores a slice. If the judge disagrees often, fix the rubric or drop the judge — do not average your way to comfort.
6. Diff versions. Same set, new prompt or model. The number that matters is the delta you can explain.

Parent commercial page: LLM evaluation services. Engagement is still QA outsourcing or QA as a service around a model swap — not a “judge licence.”

What we will not claim

• That a named foundation model is the official judge
• A public leaderboard of other companies’ products
• That LLM-as-a-judge replaces human review on high-risk answers
• Invented accuracy percentages from “our last client”

FAQs

Is LLM-as-a-judge the same as LLM evaluation?

No. Evaluation is the whole job: rubric, set, scores, version diff. A judge is one way to score some of the rows.

Can I use a cheaper model as the judge?

Sometimes, if you have measured agreement with a human on your rubric. Cheap and uncalibrated is how you automate a vibe.

Should the judge see the reference answer?

If you have a reference, prefer a check against that reference. Showing the judge the “right” answer can help — and can also leak the answer into a generous score. Test both setups on a labelled slice.

How do we start?

Book a QA consultation. Bring three examples of a good answer, three of a bad one, and the prompt you ship. We will say whether you need a judge, a mechanical metric, or a human pass.