An unverified oracle is not an answer key

Two claims underwrite almost every benchmark result, and both are usually asserted rather than demonstrated.

The first is that the answer key is right. The second, on any task described as closed-book or offline or unaided, is that the model did not have access to the thing it was supposed to be working without.

Neither claim is hard to check. Both are routinely skipped, for the same underlying reason: at the moment you would check them, everything looks fine.

The reference solution nobody ran

Here is how a task is normally authored. Someone writes the problem. Someone writes what a correct solution looks like — the reference, the golden answer, the oracle. Someone writes the graders that will decide whether a model's output is correct. Then the task ships.

Notice what never happened. The reference solution was never put through the graders.

They were written by the same person in the same sitting, so they agree in that person's head, and the artefact that would demonstrate they agree in fact does not exist. The task looks complete. Every field is populated. Review passes because a reviewer reads the reference and thinks yes, that is the right answer, which is a judgement about the reference and not about the graders.

What you learn when you finally do run it is uncomfortable and consistent. Some references fail their own graders. The failure is almost never that the reference is wrong — it is that the grader is checking something adjacent to what the task asked, or is checking it in a form the reference does not produce, or is checking a field the reference does not populate.

Which means the benchmark was not measuring the capability. It was measuring conformance to an unstated output convention that the reference happens to satisfy and a model has no way to infer. Every model that scored badly on that task scored badly on the rubric's private preferences.

Replay it act for act

The probe is simple to state: take the reference solution, replay it through the real graders exactly as a model's submission would be replayed, and require full marks.

The two words that carry the weight are real and full.

Real graders, not a validation pass written for the purpose. The moment there is a separate code path for checking the oracle, you have two graders and you are back to hoping they agree. This is the same defect that produces vacuous passes everywhere else in evaluation: two enumerations of the checks, drifting apart, each one green.

Full marks, not a passing score. A reference that scores 0.9 has told you that one of your checks disagrees with your own stated correct answer. That is a defect in the task regardless of which of the two is wrong, and a threshold below one is a mechanism for not finding out which.

The one-line version of the argument is the one we put in the blocker text itself: a benchmark whose own answer key fails its own gates is measuring the rubric, not the model.

Refuse to certify, and be precise about which state refuses

We gate certification on the provenance of the oracle, and the interesting part is that it is not a boolean.

An oracle's provenance can be verified, partial, unknown, or unverified — and only the last of those blocks a certificate. That looks backwards until you separate the states carefully:

• Unknown means nobody has recorded anything about how this oracle came to be. That is a gap in your metadata.
• Unverified means somebody looked and recorded that it has not been checked. That is a statement about the oracle.

Blocking on unknown would be blocking on missing paperwork, and would push everyone toward filling in the field rather than running the probe. Blocking on unverified blocks the one state where we have a positive assertion that the answer key was never put through the graders. Signing there would attest a score against an answer key nobody checked, which is a different and much worse thing than signing a score whose provenance we have not catalogued.

That distinction is a general one, and it is easy to get backwards in any gate: we have not recorded and we have recorded that it was not done are different facts, and only one of them is about the artefact.

The second claim: closed-book

Now the other half, where the failure is more subtle because the mechanism looks like it works.

A closed-book evaluation asserts that the model solved the task without access to some resource — the reference corpus, the rulebook, the network. It is one of the more valuable things a benchmark can claim, because open-book performance on a well-indexed corpus tells you about retrieval and very little about the model.

The usual implementation is a flag. The run is marked closed-book, the retrieval tool is disabled, the report says so.

The problem with a flag is that it is a claim about the run stored beside the run. Nothing recomputes it. If the tool was disabled somewhere else, or re-enabled by a config layer, or the corpus was still reachable by a route nobody thought of, the flag says closed-book anyway. It is exactly the sort of assertion this whole discipline exists to stop accepting.

Our version is that closed-book is not a mode of an environment. It is a different environment, derived by removing what is being withheld, and it carries a different fingerprint as a consequence. Whether a run was closed-book is then not a stored claim but a recomputable one: re-derive the withheld environment from the open one and check that it matches, byte for byte in the serialised form, the environment the run actually executed in.

Two details in that derivation earned their place. Withholding removes both the index over the corpus and the cited documents themselves — removing the index alone leaves the material reachable and only makes it inconvenient. And the check is a comparison of derived structures rather than a comparison of flags, because a flag can be set and a derivation cannot be asserted into existence.

A text filter over a command line is not a boundary

The sharpest instance of this is network access, and it is where I want to be most explicit, because our own first implementation was the wrong shape and we kept it anyway.

If a task declares that it needs no network, something has to enforce that. The easy enforcement is a filter over the command the agent is about to run: refuse anything that invokes a known network tool.

This does not work, and it does not work in a way that has nothing to do with how good your filter is. A shell is a programming language whose job is to compute the text of the command it runs. So the tool name need never appear in what you inspect. It can be assembled from escape sequences, or reversed and un-reversed, or written to a file in one command and executed in the next. Roughly:

printf '\143url --version' | sh

There is no version of the filter that survives this. The command text is not the command; it is a program that produces the command.

We kept the filter, for local development, as a way of catching the honest mistake — an agent that reaches for the network because nobody told it not to. That is a real and common case and cheap to catch.

But the docstring above it says, in as many words, that this is not a containment boundary and no future reader should treat it as one. And the certification path enforces that in the only way that counts: a run on the local backend is not eligible for a closed-book certificate. Certifying an offline claim requires a backend that enforces egress at the operating system, and a probe from inside the guest confirming that the policy was actually in force during the run.

The distinction is between a hint and a boundary. A hint that is documented as a hint is useful. A hint that is quietly load-bearing on a certificate is a lie with an implementation.

What this costs, and why to pay it

Both of these make it harder to ship a certified result. The oracle probe fails tasks that everybody believed were fine. The egress requirement means some runs simply cannot carry the claim their author wanted them to carry.

That cost is the product. A certificate is only worth something to someone who does not trust the party issuing it, and the only thing that gives it value is the population of results that were refused. If nothing is ever refused, the certificate is a receipt for a run having happened, which is a thing nobody needed a mechanism for.

Two questions for someone else's benchmark

Has your reference solution been run through your graders, and did it score full marks? If the answer is a description of a review process rather than an artefact, no.

What enforces your closed-book claim, and at what layer? If the answer is a flag, a filter, or a prompt instruction, the claim is an aspiration. Ask what would happen if the model tried, and ask who has checked.

The one-line version

An answer key that has never been run through the graders is a plausible document, not an oracle — and a closed-book claim enforced by reading the command line is enforced by nothing, because the command line is a program that computes the command.