A benchmark score also grades the harness

A benchmark result is usually presented as a relationship between two things: a model and a set of tasks.

The model solved 47 percent. The agent failed 18 of 25. Version B improved over version A.

But the number was produced by more than the model and the tasks. A runner created the starting state. An adapter translated model output into actions. A timer decided when the work ended. A retry policy decided which failures were seen again. A network policy changed what dependencies were reachable. A grader converted the final state into a verdict. An aggregator decided which attempts entered the headline.

The score grades that entire evaluation system, whether the report admits it or not.

The benchmark is two systems

The first system is the benchmark content:

• the construct it claims to measure;
• the tasks and starting worlds;
• the source and rights behind them;
• the instructions and available information;
• the intended answer or acceptable state; and
• the distribution the tasks are supposed to represent.

The second is the evaluation system:

• harness and runtime;
• model and scaffold adapter;
• observation and action translation;
• reset and teardown;
• package, sidecar, file and network setup;
• timeout, retry and concurrency policy;
• grader, judge and result parser;
• logging and missing-result behavior; and
• aggregation and exclusion rules.

An observed score is a product of both. That is not a criticism of benchmark harnesses. A harness is necessary. The mistake is treating it as neutral infrastructure after it has made decisions that affect the measured behavior.

Public projects already expose the distinction

The official Terminal-Bench repository describes a task dataset and an execution harness as distinct parts of the benchmark. The official Cua-Bench documentation defines a computer-use task through its starting state, agent interface and evaluator, not through an instruction alone. OSWorld has released verified revisions after fixing reported examples and changing execution support. The SWE-bench leaderboard exposes multiple editions and a shared agent environment for comparison.

These are useful design signals, not accusations. Serious benchmark projects already know execution is part of the object. An external audit should preserve that fact instead of importing the task files and throwing away the official runner's semantics.

A reconstruction is a hypothesis

Suppose an auditor wants to run a public benchmark inside a different sandbox provider. They reproduce the packages, mount the fixtures, translate the action API and implement the grader. The tasks all start. The final states look reasonable. The new system is called equivalent.

That conclusion arrived one experiment too early.

The reconstruction is a hypothesis about equivalence. It should be tested over exact shared task, model, scaffold and seed cells against a pinned version of the official runner. The comparison should preserve raw observations, actions, timeouts, resets, verdicts and exclusions from both paths.

When a cell disagrees, the answer is not to average the two scores. Attribute the first divergence:

• different starting state;
• observation lost or reformatted;
• action interpreted differently;
• dependency or network changed;
• timeout or retry policy changed;
• grader or parser disagreed;
• aggregation changed; or
• cause unknown.

Unknown is a valid result. “Equivalent” is not a default.

Reset is part of correctness

Many agent benchmarks depend on application or repository state. A reset that leaves one file, cache, login, database row or notification behind can make the next task easier or impossible.

The cleanest test is not a function named reset. It is an independent state measurement before and after reset, plus an adversarial sequence designed to leave residue. Run task A, mutate a state task B could exploit, reset, then measure whether B begins from its declared root.

Concurrency deserves the same suspicion. Two individually correct tasks can interfere when they share a service, account, filesystem or rate limit. A score from a parallel run is evidence about the declared concurrency posture, not automatically the same experiment as a serial run.

Timeout and retry are capability policy

A timeout looks operational. It is also a definition of success.

Does the timer include environment boot? Model queueing? Tool latency? A grader retry? Does one timed-out attempt count as failure, infrastructure exclusion or a reason to try the same seed again? Is the fastest successful retry reported, or every bought attempt?

Two leaderboards can run the same tasks and model and answer different questions because one measures success within a single bounded attempt while the other measures whether any of several attempts eventually succeeds.

Neither is inherently wrong. Hiding the policy is wrong.

The agent adapter is not the model

An adapter selects prompts, tools, observation compression, action syntax, context management and error recovery. Those are scaffold choices. A model may fail because the adapter truncated the needed observation, or appear stronger because the adapter injected task-specific strategy.

Every result should therefore bind model and scaffold identities separately. Changing the adapter creates a new experimental condition. Calling both rows by the model name erases the thing that changed.

Lossy translations need an explicit ledger. A desktop environment may expose a screen while an imported runner converts it to accessibility text. A terminal adapter may strip color, truncate output or collapse interactive state. The translation can be useful and still make the result ineligible for a claim about the original interface.

Test the evaluator with answers it should hate

Auditing tasks without challenging the grader is half an audit.

Feed the evaluation system:

• the reference solution;
• a different but valid solution;
• a no-op;
• a plausible string with the wrong world state;
• an incomplete partial result;
• a solution that edits the tests;
• an output designed to confuse the parser;
• a correct end state reached through a prohibited act;
• a timeout; and
• an infrastructure failure.

The expected verdict is not always pass or fail. Some inputs should be refused or excluded because the evaluator lacks the evidence to grade them.

Then mutate the evaluator itself. Break reset. Shorten timeout. Drop a log. Reverse one parser branch. Make one criterion vacuously pass. Change the aggregator's denominator. Seed no mutation as a control.

If the audit cannot detect those changes, its clean bill of health is a claim about code paths it never exercised.

One signed score is not enough

A mature audit should produce at least two joined artifacts.

The benchmark-quality artifact states which content and evaluation-system dimensions passed, failed, warned or were unassessable. It names every waiver and limitation.

The agent-performance artifact reports results only over the eligible measured population. It binds the exact benchmark edition, harness, adapter, runtime, model, scaffold, seeds, attempt policy, exclusions and uncertainty.

The two join by immutable identity. They do not collapse into a “trust score.” A benchmark can be reproducible and poorly aligned to its claimed construct. A grader can be calibrated while the starting state is wrong. One number hides which repair is needed.

The audit should be able to refuse the famous benchmark

Popularity is evidence of use, not validity. A public leaderboard can be valuable and still contain a broken task, mutable dependency or ambiguous aggregation policy. An external auditor should also be humble enough to refuse its own reconstruction when it cannot faithfully reproduce the official contract.

That refusal is more useful than a fresh ranking produced under conditions no maintainer or model team recognizes.

The one-line test is simple:

Can every reported difference be traced either to the agent's behavior or to a named difference in the benchmark content or evaluation system?

If not, the score may still be real. Its interpretation is not ready.