Once your evaluation dataset grows past a few hundred cases, human grading stops being practical. The obvious move is to have a strong model grade the outputs instead. It works — well enough that it is now the default in most production eval pipelines.
It also fails in specific, documented ways that will flatter your system if you do not go looking for them.
Why it works at all
A capable model given a clear rubric correlates reasonably well with human raters on many tasks. Anthropic's own evaluation guidance recommends LLM-based grading precisely because it is fast, flexible, and handles judgement calls that keyword matching cannot — tone, completeness, whether an answer actually addressed the question.
The technique that moved this from "rough heuristic" to "usable signal" is G-Eval: instead of asking for a bare score, you have the judge produce explicit evaluation steps via chain-of-thought, then fill in a structured rubric. Making the reasoning visible improves correlation with human judgement and gives you something auditable when a score looks wrong.
The three biases you must account for
Self-preference bias
Models rate their own outputs more highly than humans do. This is not folklore — it is measured. Research published on self-preference bias in LLM-as-a-judge (Wataoka et al.) found GPT-4 exhibits significant self-preference, and identified the mechanism: models assign higher scores to text with lower perplexity under their own distribution. In plain terms, a judge prefers text that sounds like itself.
The practical consequence is uncomfortable. If you use GPT-4 to judge GPT-4 outputs, your scores are inflated in a way that will not show up until a human reads the results. The mitigation is straightforward: use a different model family as judge than the one generating, and periodically cross-check with a second judge.
Position bias
In pairwise comparisons, models systematically favour one position — often whichever response appears first. This one is genuinely easy to fix: run every comparison twice with the order swapped, and treat disagreement between the two runs as a signal that the cases are effectively tied.
Verbosity bias
Longer answers score higher, independent of whether the extra words add anything. Left unchecked, this quietly optimises your system toward padding. The fix is to make conciseness an explicit rubric dimension rather than hoping the judge weighs it correctly on its own.
Calibrating a judge before you trust it
This is the step most teams skip, and it is the one that determines whether your eval numbers mean anything.
- Human-label a sample. Take 50–100 outputs and have a domain expert score them against the same rubric you will give the judge. In a Malaysian finance context, that means an actual accountant — not the engineer who built the pipeline.
- Run the judge on the same sample.
- Measure agreement. Cohen's kappa for categorical verdicts, Spearman correlation for graded scores. Below roughly 0.6 kappa, your judge is not measuring what your expert is measuring.
- Inspect the disagreements. This is where the value is. Disagreements almost always reveal an ambiguous rubric rather than a stupid judge.
- Revise the rubric and repeat. Two or three rounds usually gets you to workable agreement.
Re-run this calibration whenever you change the judge model. A provider's silent model update can shift your entire score baseline without a single line of your code changing.
Rubric design that survives contact with reality
- Score one dimension at a time. A single "quality: 1–5" score is uninterpretable. Separate groundedness, completeness, tone and format into distinct scores.
- Use binary or three-point scales where you can. Models are meaningfully more consistent distinguishing correct/incorrect than they are at 7 versus 8 out of 10.
- Define the anchors explicitly. Say what a 1 looks like and what a 5 looks like, with an example of each.
- Require the reasoning before the score. Reasoning-then-score outperforms score-then-justification, because the latter rationalises a number the model already committed to.
- Keep rubrics under version control. A rubric is part of your measurement apparatus. Changing it changes your numbers, and you need to know when that happened.
Where a judge should never be the final word
Use LLM-as-a-judge as a triage layer that flags cases for humans — not as an autonomous quality authority. Reserve human review for:
- Anything with regulatory exposure — medical, legal, financial advice, or content that falls under MOH promotional rules or BNM guidance.
- Cases where the judge and a second judge disagree.
- Anything scoring near your pass/fail threshold, where a small bias flips the verdict.
- A standing random sample — perhaps 5% of everything — to detect drift in the judge itself.
The pragmatic position
LLM-as-a-judge is the only thing that makes evaluation at scale affordable, and refusing to use it means most teams will simply not evaluate at all. Use it. Just remember that you have introduced a second model into your quality pipeline, and that model has its own failure modes.
Calibrate it against humans, use a different family than the one you are grading, swap positions in pairwise tests, score dimensions separately, and keep a human queue for the consequential cases. Do that and your numbers will mean something.
Our AI Engineering programme covers judge calibration, rubric design and evaluation pipelines hands-on — HRD Corp SBL-KHAS claimable for eligible Malaysian employers.
