Every number should resolve to a receipt

Pick any number in your last evaluation report. Now answer this, out loud, in under thirty seconds: which rows was it computed over, by what expression, and can the person reading the report run that expression themselves?

For most reports the honest answer is that the number came out of a notebook, the notebook has moved, the person who ran it has changed teams, and the population it was computed over is described in a sentence rather than in code. The number is not wrong. It is simply not checkable, which for a reader who does not already trust you is the same condition.

The gap between a claim and a measurement

A claim is a number in a document. A measurement is a number with a reproducible path back to the rows it came from.

Everything in evaluation reporting drifts from the second toward the first, and it does so through entirely reasonable steps. Somebody computes a figure correctly. Somebody writes it into a draft. The run gets extended, a few episodes are re-run, the figure moves by a point, and the draft does not. Nobody lied; the document simply stopped being downstream of the data.

The interesting question is not how to prevent that with process. It is what artefact you would have to ship for the drift to be detectable rather than merely regrettable.

Ship the data, not a description of the data

The first move is to stop describing the population and start handing it over.

For each report we build an evidence slice: a standalone, read-only database file containing the report row, the run row, every episode in the run, and every grade attached to them. Not a summary. Not an export of the aggregates. The rows.

The design decision inside that is small and load-bearing. Episodes that were re-run are copied into the slice unfiltered — every attempt, not the surviving one. It would be easy to filter them at export time, and it would be wrong, because then the rule about which attempt counts would live in our export code where the reader cannot see it. Copy everything in, and the population rule lives in the script the buyer reads and runs. If our rule for handling re-runs is bad, the buyer can see that it is bad. That is the whole point of handing over the rows.

Make the evidence executable, and don't let the caller write it

The second move is that each figure ships with the script that produces it — and the script is not something the report author supplies.

Scripts are derived from a closed registry of templates, keyed by the kind of figure. The report records which template it used; the file on disk carries the text; and the three have to agree — registry, record, file — before anything downstream will proceed. A caller cannot hand us a script, because a caller who can hand us a script can hand us one that prints the number they wanted.

Templates also take no parameters. Parameterisation is where the escape hatch always appears: one WHERE clause supplied by the author and the population is back under the author's control, invisibly, one report at a time.

And there is exactly one thing a template is forbidden to do: read the figure back out of the report row. That query would pass every time. "The number in the paper equals the number in the database" is a tautology dressed as a verification, and it is the single most likely accidental implementation of this whole idea. The script must recompute from the episode rows, or it is not evidence of anything.

Four verdicts, and the empty case named

Running a bundle of evidence scripts gives each one of four outcomes:

Note that three of those four are distinct kinds of not-verified, and keeping them apart matters. A figure that disagrees with its own query is a reporting defect. A figure whose query crashed is an infrastructure defect. They get routed to different people and they mean different things to a reader.

The bundle-level roll-up is a strict AND over the four, with one explicit extra clause: an empty bundle is not verified. In the language of most runtimes, all([]) is True, so without that clause a report that checked nothing at all would confidently report itself verified. This is the defect class that a green test suite will never catch, and it costs one line to close.

Then diff the prose against what printed

Executable evidence closes the gap between the figure and the data. It does nothing about the gap between the figure and the sentence, which is where drift actually lives — the number in the table is right and the paragraph above it still says the old one.

So the second stage takes the drafted prose and diffs every numeric value in it against the values the scripts printed.

The thing worth arguing about is how that comparison is made, and our answer is that it is a deterministic diff and emphatically not a model asked whether two numbers agree. That option is available, it is easier to build, and it is disqualified on a single ground: a verdict produced by asking a language model whether the prose matches the data is a claim from the same class of witness that wrote the prose. You would have added a step and no independence.

Deterministic comparison means you must then be explicit about what is deliberately not a claimed figure. Ours are narrow and enumerated: identifiers, parenthesised years, and cross-references to other sections. Everything else in the prose that looks like a number is required to reconcile against a closed pool of printed values.

Reconciliation is rounding, not tolerance. A prose value matches a printed value if it is that value presented at lower precision. It does not match because it is close. The moment you introduce a tolerance you have built a mechanism that accepts a wrong number, and the size of the wrongness it accepts is a parameter somebody will eventually tune upward under deadline.

Name the blind spot in the same breath

Our prose stage cannot see values written in words. A report that says just over half of episodes is invisible to it.

We say so, in the module, next to the code. Not because confessional documentation is a virtue in itself, but because an undocumented blind spot in a verification mechanism is worse than no mechanism — it produces a green check whose scope the reader will overestimate, and readers reasonably assume a checker checks the thing it is standing next to.

If you build one of these, write down what it does not look at, and put that sentence where someone reading the output will meet it.

A receipt is a pointer, not evidence

One last distinction, because it is easy to get backwards.

When a result is signed, the pointers to its evidence belong inside the signed payload. A receipt attached beside a signature is a receipt somebody could have swapped; a receipt inside it is one that provably existed when the result was minted.

But a pointer inside a signature is still only a pointer. It proves the reference was made at issue time. It does not prove the thing referred to is still there, still readable, or still says what it said.

We deliberately do not block minting on whether every receipt currently resolves, and the reason is worth stating because it looks like laziness and is the opposite. A resolution check at mint time, in the environment where the artefacts were just written, passes essentially always. It would be a gate that cannot fail — which is precisely the defect this whole system exists to detect. Resolution is a property to probe later, from outside, by a reader who does not share our filesystem. Pretending to check it at mint time would give the reader a green tick in place of a job they need to do themselves.

What to do on Monday

You do not need any of our machinery to get most of this.

Pick the three numbers in your last report that a buyer would actually act on. For each one, write the query that produces it from your raw episode rows, run it, and check it against the document. Then ship the query alongside the figure.

Two things will happen. One of the three will not reproduce — it nearly always does not, and the reason is nearly always a population definition that shifted between the analysis and the writing. And you will discover which of your figures have no expressible query at all, because they came from a step that lives only in somebody's head. Those are the ones to worry about.

The one-line version

A number in a report is worth exactly as much as the path from it back to the rows it came from — so ship the rows, derive the query from a template the author cannot write, recompute rather than re-read, and diff the prose against what actually printed.