The statistics of evaluation: what your numbers actually support

Most evaluation mistakes aren't modelling mistakes. They're claims that outrun the evidence — made honestly, by careful people, because nobody measured the noise first.

Start here: measure your noise floor

Run your identical benchmark twice. Same model, same settings, same tasks. Change nothing.

The scores will differ. Sampling temperature, tool timing, retries, non-determinism in the model itself.

Now: how much did they differ? Most teams have never measured this, which means every comparison they've ever made was read against an unknown noise floor.

The fix takes an afternoon. Run the same configuration three times. The spread is your noise floor, and it becomes the most useful number in your evaluation stack — because any difference smaller than it is not a finding. It's weather.

Two consequences. If your noise floor is four points, a model scoring three points higher has beaten nothing. And if the floor is large enough to swallow the differences you care about, that's a signal to fix the evaluation — more tasks, lower temperature, more seeds — before running another comparison.

Overlapping intervals are not a ranking

Model A scores 87%. Model B scores 84%. A is better, right?

Add the intervals. A: [83, 91]. B: [80, 88]. Those overlap almost entirely.

What you've shown is that you can't tell them apart yet. Not that they're equal — you haven't shown that either. You've shown your evidence is insufficient to rank them, which is a different and much more honest statement.

Notice how differently those two sentences land in a decision meeting. "A beat B by three points" ends the discussion. "We can't yet distinguish them" opens a better one — about cost, latency, reliability, or whether to gather more evidence.

This is why reporting intervals matters more than any other single change to how you present results. Once the intervals are visible, the honest reading becomes unavoidable — which is, frankly, exactly why point scores remain popular.

You ran twelve comparisons. One was significant.

This one catches careful people, because the mistake looks like diligence.

You run the benchmark, then slice the results. By task tier. By domain. By input length. By time of day. Twelve, twenty slices. And in one of them, a model that was unremarkable overall looks dramatically better.

Exciting. Almost certainly noise.

Test enough slices and something will look significant by chance alone. That isn't a flaw in the statistics — it's precisely what a 95% confidence level means. One in twenty crosses the line by luck. Test twenty things and expect one.

Three defences:

• Decide which comparisons matter before you run, and write them down. That single act converts most of this into a non-problem.
• Correct the threshold for how many things you tested, if you must test many.
• Treat anything found afterwards as a hypothesis, not a result — then run it again, on fresh data, as a planned comparison.

That's not bureaucracy. A large share of what gets shared as a surprising finding in this field is a slice somebody went looking for after the fact.

pass@1, pass@k, and what you're promising

pass@1: the system got it right on its first attempt.

pass@k: across k attempts, at least one was right.

pass@k is always higher, sometimes dramatically. It's a completely legitimate measure — when a human will review the output and pick the good one. In that setting, generating five candidates and having one be correct is genuinely useful, and pass@5 describes the real workflow.

It's a fantasy for an autonomous agent. In production there is no second attempt. The agent takes an action, the action has effects, and nobody is standing by to choose between five parallel worlds.

Report the metric that matches how the system will actually be used. If a human reviews: pass@k. If it acts alone: pass@1 — and resist the temptation, because pass@1 is a much less flattering number and somebody will ask why it dropped.

The honest answer: it didn't drop. You started measuring the thing you're shipping.

Where the variance actually comes from

When scores move around, the instinct is to blame the model. It's usually further up the pipeline. The candidates, roughly in order of trouble caused:

1. Which judge model you picked. Swap it and rankings can change.
2. How the judge prompt was phrased. Reordering the rubric or rewording one criterion moves scores more than people expect.
3. Sampling temperature — on the agent and on the judge. The judge's is easy to forget because it feels like infrastructure.
4. Which tasks happened to be in your set. A big variance source at small sample sizes.
5. The model itself — frequently the smallest term of the five.

The diagnostic is straightforward: hold each factor fixed in turn and re-measure. Whichever moves the score most is the one to pin hardest — version it, freeze it, treat any change as a change to the benchmark.

Teams that run this exercise usually find they can cut variance substantially without touching the model at all, just by pinning the judge and setting grading temperature to zero.

Stratify, don't just sample more

Human labels are the scarcest resource in evaluation. The question isn't how many you can get — it's where to spend them.

The default is random sampling: take 200 cases at random, label them, compute agreement. Statistically clean, and enormously wasteful, because most of those cases are ones the judge was already confident and already right about. You paid an expert to confirm something you knew.

Stratify instead. Bucket by something informative — most usefully the judge's own uncertainty — then sample thinly where it was confident and heavily where it was uncertain or where judges disagreed. Weight the results back and you get the same statistical guarantee for a fraction of the annotation.

There's a second benefit that compounds. The cases you oversample are exactly the ones that reveal where your rubric is still ambiguous. Stratified sampling doesn't just calibrate more cheaply — it generates a better queue of rubric problems to fix.

Spend your expert hours where the judge is least sure.