The grader needs a grader — and permission to refuse

Somewhere in most modern evaluation stacks there is a language model scoring the output of another language model, and the sentence describing it in the report reads something like responses were assessed against the rubric by an LLM judge.

That sentence contains an unmeasured instrument. Nobody would accept it about a physical one. If you told a regulator that lengths were assessed by a ruler, the next question would be about the ruler.

The question here is the same question, and it has a standard answer: how often does your judge agree with a human, and how wide is the interval around that number?

Percent agreement is the wrong statistic on its own

The first thing teams reach for is raw agreement: the judge and the human gave the same verdict on 78 percent of items. It sounds respectable and it can be nearly meaningless, because it does not account for the agreement you would get by chance.

If ninety percent of your items pass, a judge that says pass to everything achieves ninety percent agreement with your humans while carrying no information whatsoever. On a skewed rubric — and evaluation rubrics are almost always skewed — raw agreement is dominated by the base rate.

Cohen's kappa is the correction: agreement above what the two raters' marginal frequencies would produce by chance, scaled so that zero means chance and one means perfect. It is not exotic and it takes about fifteen lines. The reason to insist on it is exactly the reason above: it is the statistic that a degenerate judge cannot maximise by refusing to discriminate.

We hold two agreement standards, not one. Human against human sets the ceiling — if your own annotators only agree with each other at a certain level, no judge can honestly be held above it. Human against AI sits below that, deliberately, because a judge is being asked to reproduce a noisy signal and pretending otherwise just means the threshold is never met and everyone starts arguing about the threshold. On an ordinal rubric we also track within-one agreement separately from exact agreement, because a judge that is consistently one band off is a different problem from one that is randomly wrong, and the fix is different too.

The interval matters more than the point

Here is the failure mode that survives all of the above: you compute kappa on eleven items, get 0.74, and promote the judge.

A point estimate from a small sample is a number with an enormous amount of uncertainty attached, and the uncertainty is invisible in the number itself. What you need for a gate is not the estimate but the lower bound — the value you can defend, not the value you happened to observe.

The Wilson score interval is the right tool for a bounded proportion at small n, and it has a property that matters for gates: it does not produce absurd bounds near zero and one the way the normal approximation does. Promote on the lower bound and a small sample fails the gate by construction, which is the behaviour you want, because it converts we have not measured this enough into a failing gate rather than a lucky pass.

Two boundary decisions inside that implementation are worth copying.

At a sample size of zero, the interval we return is the entire unit interval. Anything from zero to one. That is the honest statement — we know nothing — and the alternative most implementations reach for, returning a zero-width interval at zero, reads to every downstream consumer as confidently zero, which is a very strong claim to make from no data.

At the same sample size of zero, the lower bound returns zero. Not because zero is an estimate, but because this bound feeds a gate that promotes judges, and zero is the direction that fails safe. The interval and the bound answer different questions and it is correct for them to disagree at the boundary.

Below the floor, refuse

Underneath all of this is a minimum sample. If fewer than some fixed number of items have been dual-labelled, we do not compute a calibration verdict at all.

The number itself is not the interesting part — pick one and defend it. What is interesting is what the system does below it, and the answer is that it returns a refusal, not a score.

This is where most implementations quietly go wrong, and the wrongness has three common shapes:

Consider the worst case concretely. A chance-corrected agreement statistic computed over a single rater has no disagreement to correct for, and the arithmetic lands on 1.0. Perfect reliability, from one annotator, by construction. That number is not wrong in the way a bug is wrong. It is wrong in the way a lie is wrong — it is precisely the number someone would fabricate if they wanted to claim their judge was calibrated.

So the rule we hold across every reliability statistic is: on degenerate input, return no value, and return a named reason alongside the absence.

Name the reason, and keep the list closed

A null is better than a fabricated number and considerably worse than a null with a name on it.

Our reasons are a closed, enumerated set, and they are specific: fewer than two raters, no pairable units, every verdict was advisory, no expected disagreement to correct against, only one category was ever used, fewer than two items, all items tied, no replicates, no per-criterion verdicts recorded, no criteria declared, the author was not in the panel.

Every one of those tells an operator something different about what to do next, which is the test a reason has to pass. Insufficient data passes no such test. It tells you to collect more of something without saying more of what.

The register is worth stating as a rule, because it generalises far beyond reliability statistics: say what is missing and what it would take. Never say unsupported. A refusal that names its own remedy is a work item. A refusal that does not is an obstacle, and people route around obstacles.

Who counts as a rater

One modelling decision that took us longer than it should have: a rater's identity is the pairing of the judge type and the specific model pin, not the judge type alone.

Collapse the pin and you have merged two genuinely different questions. Judges disagree with each other is a question about your rubric being underspecified. A judge disagrees with itself across versions is a question about your evaluation being unstable under a vendor's release schedule. Both are important, they have different owners, and averaging them together produces a number that answers neither.

Related, and equally easy to get wrong: verdicts that were marked advisory never enter an aggregate. Not weighted down, not included with a flag. Out. A statistic computed over a mixture of binding and non-binding judgements is a statistic about a population that does not exist.

Refusal is also the promotion rule

The last piece is what all this measurement is for, and the answer is that it gates a lifecycle.

A judge exists in one of three states. It starts in shadow, where it scores everything and binds nothing. It becomes calibrated when a passing calibration report exists against real dual-labelled data. Only then can it be made active and allowed to affect a verdict.

Two asymmetries hold that structure up, and they are the same asymmetry seen twice. Promotion requires a passing report to exist; there is no path to active that does not pass through evidence. Demotion is always permitted, with no report required at all — because the argument for pulling a judge out of the binding path never needs to clear the same bar as the argument for putting it in.

The same asymmetry shows up where the judge's prompt is revised in response to disagreements. A revised prompt is a new, content-addressed version rather than an edit to the existing one, and it lands back in shadow. It does not inherit the calibration of the prompt it was derived from, because the thing that was calibrated no longer exists.

And when a holdout set is used to check that the revision helped, it is evaluated exactly once — the code refuses a second evaluation outright. A holdout you can re-run is a training set with a formal-sounding name, and the number it produces after the third look is not an out-of-sample number, whatever the report calls it.

Absence is not failure

One closing distinction that keeps appearing everywhere in this work.

When a criterion goes unscored because an episode never reached the judging stage — it failed a deterministic check earlier and the judge was never asked — that criterion is not failing. It is unmeasured. Counting it as a failure understates the model; counting it as a pass overstates it; putting it in the denominator at all corrupts the rate.

So it goes in its own bucket, with its own name, and the rate is computed only over the episodes that actually reached judgement. The extra bucket is mildly annoying to carry through a codebase and it is the difference between a number that means what it says and a number that quietly means something else.

The one-line version

An LLM judge is an instrument, and an uncalibrated instrument that returns a confident number is worse than one that refuses — so measure agreement against humans with a chance correction, gate on the lower bound of the interval rather than the estimate, and when the data is too thin to support any of it, return a named reason instead of a value.