Trust that doesn't require trusting anyone

A regulator does not ask "is your model good?" Nobody who examines systems for a living asks that, because it isn't an answerable question and they know it.

Here's what they actually ask:

• How did you decide this system was fit for this particular use?
• Who signed off, and on what basis?
• What does it do when it's uncertain?
• How would you know if it degraded after deployment?
• Show me the record.

Not one of those is about model quality. All five are about process — how you decided, who decided, what happens at the edges, and whether you can prove any of it after the fact.

And every one is answerable with evaluation evidence. A frozen specification your experts approved answers the first two. Guardrail criteria and escalation behaviour answer the third. Scheduled re-runs answer the fourth. An audit trail answers the fifth.

Without an evaluation practice, all five get answered with narrative — confident, sincere narrative, which is exactly what an examiner is trained to discount.

They are not auditing your model. They are auditing your process.

Provenance, recorded at authoring time

For every task in your evaluation suite, someone should be able to answer three questions without calling a meeting:

1. Where did this case originate? Real customer data, synthesised, licensed from a vendor, public?
2. Who decided the correct answer, and when? A named person, a date, and ideally what they were looking at.
3. What are we permitted to do with it? Evaluate? Train? Move it off our infrastructure? Show it to a customer?

Record all three at authoring time, attached to the task. It costs a field.

Reconstructing provenance two years later is somewhere between expensive and impossible, and the moment you need it is always a moment when you're already under pressure — an acquisition, an audit, a customer asking whether their data was used to train something.

There's a compounding benefit. Once provenance is attached you can slice by it: which tasks came from real cases versus synthesis, and do models perform differently on those? That's a real question about the validity of your suite, and you can only ask it if the field exists.

Undocumented data is a liability wearing the costume of an asset.

Certified is not the same as exportable

A licence that permits you to evaluate on some data does not automatically permit you to redistribute it, train on it, or move it off a particular machine. These are separate rights, granted separately.

So you can land in a perfectly ordinary situation where, on the same data:

The failure mode is obvious in hindsight. Someone builds a great benchmark on licensed data, everything works, and eighteen months later a well-meaning engineer exports the task set to fine-tune a model — because nobody told them that particular action was different from the others.

The fix is to encode the licence in the system rather than in a contract nobody reads. Attach the permitted actions to the data. Make export check them and fail loudly when it isn't allowed. Then the wrong action becomes impossible rather than merely prohibited.

And a neat consequence: you can still prove what ran, using hashes, without the data ever moving.

Who is allowed to grade the grader?

If a vendor builds the benchmark, runs the benchmark, and reports the result, one party is filling three roles: rule-maker, examiner, and candidate.

That isn't necessarily dishonest. Most of the time everyone involved is acting in good faith. The problem is that it's structurally unfalsifiable — there is no observation you could make from the outside that would distinguish an honest result from a flattering one.

And crucially, nobody inside can tell the difference either. When you write the rules, run the exam, and mark your own paper, you don't experience yourself as biased. You experience yourself as thorough.

Three things restore falsifiability:

• Separate who builds the benchmark from who is measured — structurally, not by policy.
• Publish the method, not only the score, in enough detail that a competent outsider could rebuild it.
• Make results reproducible by a third party, so someone who doesn't trust you can run it and arrive at your number.

That last one is the real test, and most published results in this industry cannot survive it.

Audit trails that survive contact with an auditor

"We have logs" is not an audit trail. Logs can be edited, rotated, and selectively produced — none of which requires bad intent. It's just what logging systems do by default.

Four properties turn logs into evidence:

• Append-only, each entry linked to the one before, so removing something in the middle is detectable rather than invisible.
• Complete — and this is where most systems fail. A record containing only successful runs is a brochure. The refusals, the failed certifications, the rejected specifications are the entries that demonstrate the process was real. A trail with no refusals in it is evidence that nothing was ever gated.
• Attributable. Who or what took this action, and when. Not "the system" — the specific actor.
• Verifiable without trusting the party that holds it.

That last property separates an audit trail from a database table. If the only assurance the record is complete is your word that it's complete, the record adds nothing to your word.

The point of all of it

The goal of every practice in this piece is a result that a stranger can check.

Not a result you're confident in. Not a result produced by people who care about getting it right — though I hope that's true too. A result whose correctness does not depend on anyone's character.

What that requires, gathered in one place: the method is published in enough detail to be rebuilt; the grader was calibrated against actual people who know the work, with the evidence available; the record includes what was refused, not only what passed; the result is signed, so editing one figure breaks it visibly; and verification requires no call to whoever issued it.

None of that makes anyone honest. That's the point. It makes honesty checkable — and checkable honesty is the only kind that survives contact with incentives, deadlines, and the completely ordinary human wish for the number to be good.

Don't ask to be trusted. Make trust unnecessary.