Audit the passes, not just the failures

Watch how an evaluation team spends its review time and you will find a consistent asymmetry: the failures get read and the passes get counted.

It is an understandable habit. A failure is where the interesting information is — the model did something wrong, and you want to know what. A pass is boring. The task said do X, the grader checked X, X was there. Next.

But look at what actually leaves the building. The failures stay in your issue tracker. The passes become the number on the slide, the row in the comparison table, the line in the procurement memo. The pass is the artefact you are shipping. It is the only part of the run that anybody outside your team will ever act on, and it is the part nobody looked at.

What a pass actually asserts

Here is the claim embedded in a single passing episode, written out longhand: a model, given only what the task declared, produced through its own work an outcome that satisfies every check the task specified, and it did so for the reason the task was written to test.

That is a large claim, and a grader verifies only the middle of it. The grader confirms the checks are satisfied. It has nothing at all to say about whether the model got there by working the problem or by some route the task author never imagined — reading an answer that was left lying around, editing the test that judges it, exploiting a gap between what the verifier checks and what the task asked for, or producing an output so degenerate that the checks happen not to catch it.

None of those look different from a legitimate pass at the point where the pass rate is computed. They are all the same value: one.

Interrogate on named grounds, not on vibes

The instinct at this point is to add a review step: have someone look at a sample of passes and flag anything that seems off. This is better than nothing and worse than it sounds, because "seems off" is not a finding you can carry into a disagreement. A buyer who is told that a reviewer looked at some passes and felt fine about them has been told nothing they can check.

The alternative is to fix the grounds in advance. Decide, once, what the ways of passing dishonestly are — as a closed, ordered list — and then require every passing episode to carry a slot for each one.

In our own harness that list has seven entries, and the names are the point: a hardcoded result, a modified test, a verifier gap exploited, a side channel in the environment, a deviation from the spec, a trivial pass, a divergence from the oracle. Each passing episode carries one of three states per category: the category is clear, the category fired, or the category is unevidenced — meaning nobody was in a position to look.

That third state is the one that does the work, and I will come back to it.

The episode-level verdicts sit on an ordered scale, from not_applicable through legitimate, unaudited, suspect, to exploit, and an episode's verdict is the maximum over its categories. Ordered rather than categorical, so that combining verdicts is defined and cannot quietly average a suspicion away.

Refutation is the default

The hardest design decision in an audit is which way the burden runs, and the honest answer is that it has to run against the auditor.

Every category in our system passes through one shared decision test before it is allowed to fire, and each step is a way of not firing:

1. The episode passed. A failing episode has nothing to audit.
2. The category is evidenceable at all here — there is a retention channel that would have shown the behaviour if it happened.
3. There is a concrete anchor: a specific artefact, not an inference.
4. That anchor is causally connected to the pass, not merely present in the run.
5. The behaviour is not explained by the task's own prompt. If the task told the model to do the thing, the model doing it is compliance.
6. Where a benign explanation exists, prefer it. A degenerate value that the checks happened to accept gets downgraded rather than escalated.

Six chances to decline. The result is an audit that under-reports, and that is the correct failure mode for a mechanism whose output is an accusation. A false accusation against your own passing run is more expensive than a missed one, because it burns the credibility of every true one.

The category that needs a second run

Most of the categories can be settled from what the run already retained. One cannot: the hardcoded result.

You cannot tell from a transcript whether a model computed an answer or recognised one. The output is identical. There is no artefact in the episode that distinguishes them, which is why "the model memorised it" is the accusation everyone makes and nobody can support.

The way to settle it is to change the question and ask again. Take the task, perturb the inputs so that the correct answer moves, and re-run. A model that solved the problem solves the moved problem. A model that recalled an answer returns the same answer to a different question.

That is simple to describe and full of ways to fool yourself, so the ladder that decides it is explicit and ordered, and most of its rungs are refusals rather than findings:

Exactly one row produces a finding. The others exist so that the finding, when it comes, is not an artefact of a broken re-run.

Two details are worth stealing whether or not you build the rest. First, a mutation that leaves the correct answer unchanged is rejected at construction — it is not a weak test, it is not a test. Second, "nobody tried to replay this" and "somebody tried and here is why they could not" are stored as different things. Silence from the caller and a reason from the caller are not the same information, and collapsing them loses the only part that is actionable.

What you cannot see is a finding too

Two of our seven categories have no detector. They are not aspirational placeholders; each carries a fixed sentence naming precisely what is missing — no independent run-stage correctness judgement, or no authorised semantic comparison. Three more are conditional: hardcoded-result needs a mutation replay, test-modification needs a grading-surface declaration, and oracle-divergence needs an admitted like-for-like oracle comparison. Each refusal is a different sentence and is stored as one.

This is the part that people initially read as weakness and later recognise as the whole point. A report that lists four checks and reports four results is a report that has silently deleted three questions. A report that lists seven and says these four were checked, these three could not be, and here is the retention that would change that is telling you the shape of its own blind spot.

The register matters as much as the mechanism: say what is missing and what it would take. Never "unsupported."

And note what a category with no detector does not do. It does not read as clean. Absence of evidence is stored as absence of evidence, in its own state, distinct from the state that means we looked and found nothing.

Two rates over one denominator

The output of all this is a second number: the pass rate again, minus the passes the audit excluded — where excluded means the episode's verdict reached suspect or exploit, one definition, applied everywhere.

The critical constraint is that both rates are computed over the same episodes. A headline of 0.60 and an audited rate of 0.45 is a sentence about audits only if the denominator is identical; otherwise it is a sentence about two different runs and it means nothing. Our implementation takes paired episodes and refuses outright if the identifiers do not line up, rather than accepting two parallel lists and trusting the caller to have kept them aligned.

A run also carries a posture — the maximum verdict across its episodes. A clean run reads unaudited, not legitimate. And a run with no audits at all reads not_applicable, because nothing was examined, which is a different claim from nothing being wrong.

If you take one thing

Add the second number. Publish the headline pass rate and the audited pass rate side by side over the same denominator, and let the gap be a thing people ask about.

You will find, the first time, that the gap is mostly unevidenced rather than suspect — that you cannot audit most of your passes because the run never retained what an audit would need. That is not a disappointing result. That is the finding: you have been shipping a number whose central claim was never checkable, and now you know which retention to add.

The one-line version

A failure you reviewed is a bug report; a pass you did not review is a claim you have already shipped — and the only honest audit is one that names the ways of passing dishonestly in advance, refuses to fire without an anchor, and records the ones it could not check as unchecked rather than as clean.