Benchmarks with a date on them

Note: the figures in this post are illustrative examples, not measured results. The March/June table shows the shape of the effect, not a measured decay.

Here's a question a bank actually wants answered: given this morning's filings and the current curve, does this credit still clear our overlays?

You cannot ask that with a made-up fixture — the whole point is that it's about the real world at a real moment. And you cannot ask it with a live API call either, because the moment your task reads the live internet, two runs of it are no longer comparable.

That looks like a dilemma. It isn't.

Why live calls break measurement

Run the benchmark in March, score 90%. Run it in June, score 81%.

Did the model get worse, or did the world get harder?

You cannot tell. And a benchmark that can't tell you that isn't measuring the model — it's measuring the model and the world, summed, with no way to separate them. Every conclusion you draw from the change is unsupported.

There's a second problem. A live call means the benchmark isn't replayable. Six months later, when someone asks why the March number was what it was, the evidence is gone. The pages changed.

Capture and replay are separable

The insight that dissolves the dilemma: touching the network and reading the network don't have to happen at the same time.

Touch the network exactly once, in advance, at a named instant, and record what came back. The agent then reads that frozen capture through an ordinary tool interface and has no network access at all.

The world is as frozen as it ever was in a synthetic fixture. What's new is that it has a date — and the date can be recorded and signed alongside the result.

Five rules that make a capture honest

No live egress at run time. The agent reads the capture, never the network. If it can reach out, everything below is unenforceable.

No lookahead. Not one record in the capture may postdate the instant the world was cut at. This is the subtle one and the most damaging when violated: eleven hours of hindsight is the difference between a benchmark and a crystal ball, and it flatters every model equally so nobody notices.

Point-in-time faithfulness. A restated figure is not what was knowable then. Financial data in particular gets revised — if your capture contains the revised version, you're testing whether a model can use information that didn't exist at the moment you claim to be testing.

Provenance and rights. Record where each captured item came from and what you're permitted to do with it. Licensed data may be evaluatable but not redistributable, and those are separate rights.

Replay determinism. Read the same capture twice; the bytes must be identical. If they aren't, you don't have a capture, you have a cache.

Each of those should fail loudly rather than warn. A rule that produces a log line nobody reads is not a rule.

What this unlocks: decomposing a score change

Once captures are dated and recipes are fingerprinted, you can do something otherwise impossible.

Hold the recipe completely identical — same tasks, same rubric, same graders, same judge — and move only the capture forward three months. Re-run the same model.

Nine points, and every one of them is the world, not the model.

That's a number nobody in the industry can currently produce, and every organisation with an agent in production has the question. "Our accuracy dropped this quarter" is currently answered with speculation. It doesn't have to be.

The reverse decomposition is just as useful: hold the capture fixed and move the model. Now the delta is entirely the model, and it's the honest fine-tune delta.

One variable at a time. It's the only rule, and captures are what make it possible for time-dependent work.

Practical notes

Capture more than you need. Storage is cheap and re-capturing a past instant is impossible. Capture broadly at the named moment, select narrowly at build time.

Version the capture, not just the task set. A benchmark on the March capture and the same benchmark on June's are two benchmarks. Label them that way, or someone will compare them and report the difference as progress.

Watch for silent staleness. A capture that's a year old may test a world nobody operates in any more. Freshness is a property to schedule, not to assume — plan re-captures the way you plan re-calibration.

Check the boundary carefully. The lookahead rule is easy to violate at the edges: a document dated before your instant that was actually published after it, a database row with a backdated timestamp. If you can, verify against publication time rather than record time.

The one-line version

A benchmark that reads the live world can't be reproduced. A benchmark that reads a made-up world can't answer the question.

A benchmark that reads a real world, frozen at a named and signed instant, does both — and it's the only way to tell whether the model got worse or the year did.