Test the grader, not just the model

Everyone accepts that the model under test needs rigorous evaluation. Almost nobody applies the same standard to the thing doing the evaluating.

Your grader is production code. It decides which models you ship, which vendors you pick, and what number goes in front of your board. A defect in it is worse than a defect in the model, because a bad model produces bad outputs you can see, and a bad grader produces good-looking numbers about a bad model, which you cannot.

And it is unusually hard to test well, for a specific reason.

Why example-based tests miss grader bugs

The way everyone tests a grader is by example: build an episode that should pass, assert it passes; build one that should fail, assert it fails.

That approach is structurally blind to the most dangerous class of grader bug, because every example you write is an example of a case you already thought about. The bugs live in the cases you didn't: the empty one, the malformed one, the one where two configuration options interact, the one where a task declares something that nothing reads.

We watched a suite of ~4,000 example-based tests fail to catch nine consecutive instances of the same defect. Not because the tests were bad — they were good tests of the behaviour they described. The defects were in the space between them.

Properties cover that space. An example says "this input produces that output." A property says "for every input in this space, this must hold." You then generate the space.

Five properties worth stating

These are not exotic. They're the things you already believe about your grader and have never written down.

P1 — An agent that does nothing never scores as correct.

Construct the null episode: no tool calls, no submission, no changes to the world. For every task configuration your system will accept, that episode must not come back correct.

This is the single highest-value property in the set. It catches the entire "empty collection of checks" family, it takes twenty minutes to write, and in our case it found a defect present in both of our graders simultaneously — which meant no amount of cross-checking one against the other would ever have surfaced it.

P2 — Every claim a task makes is read by something that can fail it.

If a task declares an expected answer, some check must consume it. If it declares an expected end state, likewise. A declared expectation that nothing reads is not a lenient check — it's a claim that silently isn't a claim, and the task will pass regardless of what the agent does about it.

The right response to violating this is refusal, not failure. The task is defective, not the model.

P3 — Anything you call ready must survive your own quality checks.

If your pipeline can label a task "complete," "validated," or "promotable," that label has to be verified against the checks rather than asserted. We found tasks emitting a full-completeness status that were rejected 100% of the time by the very next stage. The status and the gate had drifted apart, and nothing compared them.

P4 — If you have two graders, they agree — or both refuse, for the same reason.

Most mature evaluation systems end up with two: one that runs at authoring time to validate the task, one that runs on live results. They start identical and drift, because they're fixed at different times by different people.

Where both can score an episode, the verdicts must match. Where neither can, both must refuse and give the same reason. This property is worth its cost on its own — but note its blind spot: it cannot see a defect that exists identically in both. Ours did. P1 caught what P4 could not.

P5 — A strictly better episode never scores lower.

Take any episode. Add one more correct action. The score must not go down. This is cheap to generate — you already have the episodes — and it catches an entire family of ordering, short-circuit and accumulator bugs that no example test targets, because nobody writes an example for "and then it did one more right thing."

The step almost everyone skips

Write the properties, run them, watch them pass, feel good. This is the moment the whole exercise fails.

A property test that passes when its invariant is broken is worse than no test at all, because it converts an unexamined risk into a documented assurance. You now have a green check next to a defect.

So verify the verification: deliberately break each check the property is supposed to protect, and confirm that the named property fails. Not "some tests fail." The specific one, by name. Then restore it.

Two practical notes, both learned expensively:

• Clear your build caches between the break and the restore. Stale compiled bytecode will hand you a false result and you'll draw exactly the wrong conclusion from it.
• Record the results in a table — which check you broke, which tests failed, whether that matched what you predicted. The mismatches are the interesting rows. In our case one mutation caused nothing to fail, which told us our generated space was missing an axis. That gap was a live bug: a state check that compared half of what it claimed to, returning correct rather than failing.

Generating the space

You do not need a fancy property-testing library, though use one if you have it. For most evaluation systems the configuration space is small and enumerable, and a seeded exhaustive sweep beats random search — it's deterministic, it's reproducible, and a failure gives you the exact configuration rather than a seed to re-run.

The axes to vary are the ones your task format actually offers:

• which kinds of check the specification declares
• which of those are in the scored set versus advisory
• whether the task overrides anything per-instance
• where the task came from — hand-authored, generated, derived from real traffic
• the shapes where a claim is declared in one place and expected in another

Multiply them out and assert the count in a test, with a message that names each axis and what it's for. Otherwise someone shrinks the space during a refactor, every property below becomes a tautology, and the suite stays green forever.

Bound the total. These run on every commit. Ours settled at a few tens of thousands of configurations in under a minute; if yours is heading for millions, sample deliberately and say what you sampled.

The one-line version

You test your model against your grader. Nothing tests your grader — so state the five things you already believe about it, generate the space, and then break each check to prove the properties can see it.

The properties you can't break are the ones you don't have.