Can you trust an AI to grade an AI?

At some point every evaluation programme hits the same wall. Human grading is the gold standard and it does not scale. So: can you use a model to grade a model?

Yes — but only if it has proven it agrees with people who know the job. Almost everything interesting is in what "proven" means.

Why "90% agreement" usually means nothing

Someone tells you their judge agrees with human experts 90% of the time. It sounds strong. It usually isn't.

Suppose 85% of your cases are passes — which is common, because most work is done correctly. A judge that says "pass" to everything is right 85% of the time. It has learned nothing. It agrees constantly.

So 90% isn't 90% of skill. It's five points above a rock with the word "pass" written on it.

The habit worth building: whenever you see an agreement figure, ask what it would be under guessing. If nobody can tell you, assume it's uncomfortably close to the number you were shown.

What kappa corrects for

Cohen's kappa asks one question: how much better than chance is this agreement, given how the answers are actually distributed?

It takes observed agreement, subtracts the agreement two parties would reach by guessing at the same base rates, and scales what's left.

Rough reading:

The base-rate-riding judge from the previous section scores near zero kappa regardless of how impressive its raw agreement looks. That's the whole point of the metric.

Kappa is agreement minus luck.

Why a point estimate is not evidence

Here's a mistake that survives in production far longer than it should.

You calibrate on 40 cases. Agreement comes out at 0.82. It goes in the deck.

But a point estimate from a small sample is mostly luck. Run it on a different 40 and you might get 0.68. Nothing changed except which cases you drew.

The honest question isn't "what was the agreement?" It's: given this much evidence, what's the worst the true agreement plausibly is?

That's a confidence lower bound. A Wilson lower bound is the standard choice for proportions and behaves well at small samples, where the naive interval falls apart.

Now gate on the lower bound rather than the point estimate, and something elegant happens: a small lucky sample can no longer buy its way past the floor. Fewer cases widens the interval, which pushes the bound down. To clear the bar you have to bring more evidence.

Small samples don't produce weak evidence. They produce confident noise.

The three biases you should test for before trusting a judge

Position bias. Show a judge two answers and it tends to prefer whichever came first — or consistently second. The content didn't change, only the order.

Verbosity bias. Longer answers score higher. Judges mistake length and confidence for quality, much as tired human markers do.

Self-preference. A judge tends to favour output from its own model family. If you evaluate five vendors using one vendor's model as the judge, there is a conflict of interest sitting in the middle of your methodology.

All three are cheap to test:

• Swap the order and re-run. Does the verdict flip?
• Pad a wrong answer with confident filler. Does the score rise?
• Judge the same set with a different model family. Does the ranking hold?

Three experiments, an afternoon each. A bias you haven't tested for is a bias you're shipping.

Calibration is not a one-time event

When you calibrate, you measure agreement against one distribution of cases, using one model version, at one moment. Change any of the three and your evidence is stale.

Three things move underneath you:

• The provider updates the judge model. The dangerous one, because it arrives without a changelog anyone reads, and your scores shift while the thing being evaluated hasn't changed.
• Your input distribution shifts as usage grows and you see cases the calibration set never contained.
• Someone edits the rubric. Slightly. For clarity. Which is a new rubric.

Two defences: pin the judge version explicitly so an upgrade is a decision rather than an event, and re-check agreement on a schedule rather than on a hunch.

The operational question isn't "is my judge accurate?" It's "would I know if it stopped being accurate?"

One judge is usually the wrong shape

"Was this done correctly?" feels like one question. It's at least six: did it reach the right outcome, was the process legitimate, did it stay inside the rules, is the output well-formed, was the cost acceptable, is every claim grounded in retrieved evidence?

Ask a single judge for a single verdict and it will trade those off against each other silently.

Score them separately and require all to pass — conjunctive, not averaged. The reason is what averaging does at the edges: a beautiful, well-argued answer that quietly violated a hard rule can outscore a plain answer that followed every one. In a regulated workflow that isn't a scoring quirk, it's the entire risk.

There's a practical bonus. When something fails, you know which of the six refused it. That's a failure taxonomy, for free.

The rule underneath all of it

A judge that hasn't cleared a floor against people who know the work isn't a measurement instrument. It's a second opinion with unusually good grammar.

Agreement is proven, not assumed.