How to write an evaluation that holds up

Most evaluations get scoped from capability: what can the model do? Let's enumerate the skills and test each one.

It feels systematic. It produces enormous, unfocused suites that measure everything and decide nothing.

Scope from consequence instead

Ask a different question: what is the mistake we would have to explain to someone?

• The wrong approval that reaches a customer.
• The exception nobody documented.
• The confident answer with no source behind it.
• The escalation that should have been handled.

Three or four sentences like that are a better specification than any capability taxonomy, for a simple reason: each names a failure that costs something, so each justifies its own test. You will never have to defend why that case is in the suite.

It also sets your difficulty distribution for free. The cases you're afraid of are, by definition, the hard ones — which is where measurement resolution actually lives.

Four tiers, and the trap

Every suite needs four kinds of case:

• Clean — the ordinary case, done right. Catches catastrophic regressions, sets your baseline.
• Thin — information missing, still decidable. A file without one document; a request with an ambiguity a professional would resolve rather than escalate.
• Exception — the rule doesn't apply cleanly. Handling it correctly requires knowing why the rule exists.
• Adversarial — designed to mislead. The obvious answer is wrong, or surface features point one way and correct handling points another.

Here's the trap. Most suites end up 90% clean cases, because clean cases are the easiest to write and the easiest to get experts to agree on. And clean cases are exactly where every serious model already succeeds.

Your resolution comes almost entirely from the last two tiers. If you're short of time, build fewer clean cases — not fewer exceptions.

Edge cases aren't rare. They're the point.

"That case only happens 2% of the time — is it worth testing?"

Yes. More than the common case.

The common path is where automation is easy. If the model handles it, good — but that saving is already priced into the business case, and every serious model handles it. Testing it heavily tells you almost nothing.

The 2% is where a human currently has to stop and think. It's why the role requires judgement. It's the expensive part, and it determines whether you can actually remove the human from the loop or merely reduce their volume. Those are completely different outcomes commercially.

So weight by consequence, not frequency. A frequency-weighted suite flatters the model. A consequence-weighted suite scores lower and tells you something you can act on.

If the score drops when you rebalance, that's not the model getting worse. That's you finally measuring the part that mattered.

Nobody hired a human to do the easy 98%.

How many tasks do you actually need?

It depends entirely on what difference you need to detect. Rough figures for a binary pass rate near the middle of the range:

Precision improves with the square root of the sample. To halve your uncertainty you need four times the tasks — so the fourth doubling costs the same effort as the first and buys a quarter as much.

Which flips the question usefully. Don't ask how many tasks you can afford. Ask what size of difference would change a decision you're going to make, then buy exactly enough resolution to see it. Anything beyond that is precision nobody will act on.

Writing the oracle is where you find out you disagree

The oracle is the known-correct answer for a task, written by someone who could actually do the job — not generated, not inferred from a label. Written.

Here's what happens the first time you ask two qualified experts to write one for the same case: you regularly get two different answers.

The instinct is to treat that as a nuisance. Pick one, move on, there's a deadline.

It's the most valuable signal in the entire build. If two people who do this work for a living produce different correct answers, your specification is underspecified — and that gap was going to hurt you later, when a model landed in the middle of it and you had no principled way to score it.

So use it as a test. Two experts, same case, independent oracles. If they agree, your rubric is real. If they don't, you've found an ambiguity for the price of one afternoon.

Disagreement found at authoring time is free. The same disagreement found after you've scored four hundred episodes costs you the whole run.

Guardrails are criteria too

Most rubrics describe what a good answer contains. Almost none describe what must never happen.

• Never write to a system of record during a read-only task.
• Never proceed on an unverified identity.
• Never assert a figure it did not retrieve.
• Never escalate without recording why.

These aren't quality criteria — they're prohibitions, and the crucial part is how you score them. Not as points. As hard gates. A violation fails the episode outright, no matter how good everything else was.

Consider what happens if you average them in. An agent that produced a beautiful, correct answer — and along the way wrote to a production system it had no business touching — still scores well. The violation becomes a small deduction inside a large positive number and disappears.

In a regulated workflow that's not a scoring quirk. That's the entire risk you were asked to evaluate, quietly averaged away.

Some failures shouldn't be survivable by scoring well elsewhere.