What a benchmark actually is — and why yours probably isn't one

Almost every company deploying AI has something they call a benchmark. Almost none of them have one.

That isn't a criticism of the people building them. It's a definitional problem, and the definition matters more than it sounds.

The definition

A benchmark is a fixed set of tasks, with answers decided in advance, used to compare systems fairly.

Three properties, and all three are load-bearing:

1. The same tasks for everyone. If two systems saw different questions, you compared two different exams.
2. Answers settled before measurement. If the correct answer was decided after seeing the output, you didn't measure — you rationalised.
3. A score that reproduces. If running it again next week gives a materially different number for the same system, the number wasn't about the system.

Remove any one and you still have something useful. You just don't have a benchmark. You have a demo — a sample of one, selected by someone with an interest in the outcome.

How internal evaluations lose each property

The failures are specific and predictable.

Property one goes when the evaluation set grows organically. Someone adds cases that a model failed last month. Reasonable — except now the suite is biased toward one system's weaknesses, and a competitor evaluated on it is sitting a different exam.

Property two goes when grading is done by whoever is available, case by case, without a written rubric. Two graders, two standards. Worse: the same grader on Monday and Friday is two standards, and neither of them is written down where you could check.

Property three goes silently and is the one almost nobody measures. Run your identical evaluation twice — same model, same settings, same tasks. The scores will differ. Sampling temperature, tool timing, retries, model non-determinism. How much do they differ? If you don't know, every comparison you have ever made was read against an unknown noise floor.

That last point is worth an afternoon of your time. Run the same configuration three times. The spread is your noise floor. Any difference smaller than it is not a finding — it's weather.

Why "a ruler that doesn't move" is the whole idea

The purpose of a benchmark is comparison. Comparison requires that the instrument stay fixed while the thing being measured changes.

This sounds obvious and is violated constantly, usually with good intentions:

• You fix some broken tasks between two runs, because they were broken and fixing them was right. Now part of your improvement is the benchmark becoming fairer.
• You add harder cases because the old ones saturated. Now your score dropped and nobody can say how much was the model.
• Your judge model was updated by its provider. Now you're reading drift as progress.

Each of those is a defensible decision. Together they mean your two numbers aren't comparable, and nobody in the room will notice, because both numbers came from "the benchmark."

The discipline that fixes this is unglamorous: version the benchmark like you version code. A changed rubric is a new version. A changed task set is a new version. A changed judge is a new version. And a comparison across versions is not a comparison — it's two facts that happen to be adjacent.

The uncomfortable implication for public leaderboards

Everything above applies to public benchmarks too, and they have three additional problems.

Saturation. When every serious model clusters at 92–95, the gaps between them are smaller than measurement noise. You're still getting different numbers. You're reading variance.

Contamination. Public benchmarks are published. Published text gets scraped. Scraped text becomes training data. Nobody has to cheat for this to happen — it's the default behaviour of the pipeline. A model that has seen the exam isn't being tested; it's being asked whether it remembers. Those produce similar scores and very different production behaviour.

Relevance. Public benchmarks measure general ability on general tasks. Your company's actual work isn't on the list. Nobody's is.

None of this makes leaderboards useless. Knowing a model is generally capable is genuinely worth knowing, and it narrows a shortlist. It just isn't the question you're actually asking, which is whether this system can do a specific job at your company under your policy.

General ability is not job performance. The gap between those two things is where post-deployment surprises live.

What to do on Monday

If you want to move from a demo to a benchmark, three steps, in order:

1. Write the rubric down. Not "was this good" — specific criteria a second person could apply and reach the same verdict. If two of your experts disagree when applying it, it isn't finished.
2. Freeze the task set and version it. Anything you add creates version two. Keep version one runnable.
3. Measure your noise floor. Three runs, same config. Now you know what size of difference is real.

None of that requires a platform, a vendor, or a budget line. It requires deciding that the number is supposed to mean something.

A benchmark is a ruler. Rulers only work if they don't move.