The scorekeeper is an attack surface

A reward is not a fact about whether an agent succeeded. It is a program that turns selected evidence into a number.

That program may be thoughtful, reviewed and covered by tests. It may still reward the wrong behavior. A model can find the gap deliberately during reinforcement learning or accidentally through ordinary optimization. Either way, the measurement becomes the target.

The usual response is to add a better judge. That helps only if the new judge is harder to exploit, and if its additional judgment does not create a different shortcut. A stronger response is to treat the entire evaluation system as an attack surface.

The evaluator needs an immune system.

The scorer is part of the environment

An agent does not interact only with a task. It also interacts with whatever determines whether the task counts as complete.

That includes:

• deterministic checks;
• reference answers and hidden tests;
• model judges and their evidence views;
• parsers and format requirements;
• timeout and retry behavior;
• aggregation and exclusion rules; and
• the feedback exposed after an attempt.

Each component leaks a little information about what the system values. Repeated optimization can turn that information into a strategy that scores well while leaving the underlying work undone.

This is not limited to obviously adversarial agents. If a grader overvalues fluent summaries, ordinary training will produce more fluent summaries. If a hidden test accepts a file with the right name, the model will learn that the name matters more than the contents. If a judge never inspects tool evidence, plausible unsupported claims become locally optimal.

An exploit needs two measurements

Calling every surprising pass “reward hacking” makes the term useless. An exploit needs at least two measurements:

1. The candidate improves the evaluator-facing score.
2. The candidate preserves or worsens the underlying task outcome under an independent validity check.

The second measurement is the one most demonstrations omit. A clever answer that satisfies an unusual criterion may be a legitimate alternative solution. A shorter trajectory may reflect efficiency, not gaming. A judge disagreement may be a measurement defect, not an exploit.

The test therefore needs a shadow evaluator whose evidence and failure modes differ from the attacked scorer. It may combine held-out deterministic checks, a separately admitted model judge, blinded expert review, or a counterfactual world that exposes whether the claimed action had its intended effect.

No single shadow measure settles every case. The important property is independence from the surface being attacked.

Attack in levels, not anecdotes

A useful wind tunnel increases the attacker's information and capability in declared levels.

Level zero: controls. Run valid solutions, clear failures, infrastructure faults and ambiguous cases. The evaluator should reward, reject, exclude or abstain exactly where declared.

Level one: black-box search. The attacker sees task inputs and scalar outcomes, but not grader internals. Search for score improvements that do not improve the shadow outcome.

Level two: typed evaluator knowledge. Expose the public interface, output schema, named criteria and allowed feedback. This tests the information an optimizing agent could reasonably infer through repeated use.

Level three: implementation-aware attack. Where the threat model permits it, expose code or detailed grader behavior and search for direct weaknesses. This is useful for public benchmarks and internal red teams. It is not a fair model-capability comparison unless every participant gets the same access.

Every level needs an attack budget: attempts, tokens, wall time, tools and feedback. “No exploit found” without a budget is not a finding. It is a stopped search.

A natural exploit is more dangerous

Some attacks produce bizarre strings no real agent would emit. They prove a parser bug, but may say little about normal optimization pressure.

Report naturalness separately from exploitability. Ask whether the candidate:

• resembles a plausible work product;
• follows the declared interaction protocol;
• transfers across paraphrases and seeds;
• survives small presentation changes; and
• appears under ordinary agent search rather than only a handcrafted attack.

An unnatural exploit still deserves repair when the scorer is public or reused for training. It should not be presented as evidence that deployed agents will discover it unaided.

Localize the first causal defect

A failed trajectory may contain many bad events. The useful question is not which event looked worst. It is which earliest change would have prevented the invalid reward.

Freeze the world before each candidate point, apply one controlled mutation, and replay the remainder. If removing one misleading observation makes the exploit disappear while every unrelated state root stays fixed, the attribution is stronger than a retrospective explanation.

This produces a causal map:

• origin: the first evaluator or environment defect required by the exploit;
• propagation: later steps that carry it forward;
• symptom: the visible bad result;
• independent fault: another problem not on the same chain; and
• unresolved: evidence is insufficient to distinguish the alternatives.

The unresolved label matters. A confident story written after the fact is not causal evidence.

Repair, then attack the repair

The obvious patch often creates a new failure.

A stricter exact-match check blocks a semantic shortcut and rejects valid phrasing. A more verbose judge rubric closes one exploit and adds position bias. A new hidden test catches hardcoded output and accidentally reveals the answer through its error message.

The repair loop should therefore run in four stages:

1. Freeze the original exploit and an honest control set.
2. Apply one versioned patch.
3. Re-run the exploit, controls and held-out tasks.
4. Give a fresh attacker the same budget against the patched evaluator.

The patch is admitted only if it reduces the targeted exploit without worsening honest controls beyond a declared tolerance. Call that the autoimmunity check: the evaluator must not protect itself by attacking valid work.

Keep capability and evaluator findings separate

The final report needs at least four dispositions:

• confirmed model failure;
• confirmed evaluator defect;
• infrastructure exclusion; and
• unresolved attribution.

Collapsing them into pass and fail contaminates every downstream use. A model team retrains on harness defects. A benchmark owner celebrates an exclusion as a difficult task. A customer sees a confident percentage whose denominator contains different phenomena.

The report should preserve the evidence packet for each disposition: exact task edition, runtime, observations, actions, scorer result, shadow result, attack budget, mutation, replay roots and reviewer decision.

The immune system is continuous

An evaluator that survived yesterday's attacks is not permanently valid. Models change. Scaffolds change. Tool access changes. A public benchmark teaches the world how it scores.

The wind tunnel belongs in the release loop. Re-run the admitted attack battery when the evaluator, runtime, model family or task distribution changes. Add newly observed shortcuts as regression cases. Keep a hidden vault to test whether the repair generalized beyond the examples used to create it.

The strongest outcome is not “unhackable.” No finite experiment earns that word.

The defensible outcome is narrower and more useful:

Under this frozen threat model, evidence contract and attack budget, these exploit families were detected, these repairs survived honest controls and held-out re-attack, and these risks remain open.

That is how a score becomes evidence instead of a target-shaped promise.