The Reward‑Hacking Wind Tunnel

When a lab reports “94.2 on MATH,” nobody checked 94.2% of the answers. A scoring pipeline did: an answer extractor decided what the model “answered,” a judge or rubric decided whether it was right, a unit-test verifier decided whether the code worked.

Two earlier pillars of the Gradia program made the rest of the evaluation stack honest: Interruptible Universes proved what happened in a changing world, and Gradia Guard made the runtime record tamper-evident. The scorekeeper is the last trusted component — and it is an attack surface. Prior work already showed how wide open it is: a GSM8K judge whose pass rate could be pushed from 0.72 to 0.94 while true accuracy stayed at 0.20, and multi-judge panels that agree with each other at Fleiss κ = 0.151, barely above chance.

The trick that makes the whole study cheap: start from a verified-correct answer, apply a typed transform that breaks the truth in a machine-checkable way, and let a deterministic oracle (numeric equality, an answer key, a hidden test suite) confirm the result is wrong. If the judge still says PASS, that is a confirmed exploit — wrongness is double-locked, and no expert ever looks at anything.

Attacks run on four rungs, never pooled: L0 natural incidence, L1 a fixed metamorphic catalog, L2 budgeted adaptive search against the judge, L3 patch-aware re-attack. A naturalness constraint bans injection artifacts — every exploit must be an answer a deployed model could plausibly emit (an automated screen passed 100% of an 80-exploit stratified sample). That line is what separates a reward-hacking study from a rediscovery of prompt injection.

Public benchmarks can tell you a model is being gamed. Only a branchable, witnessed universe can tell you which counterfactual causes it — because only there can you change one fact and prove you changed nothing else.

A fork on the Gradia engine snapshots a parent world, applies an intervention that changes exactly one declared variable, and returns content-addressed before/after world roots plus a hash-chained occurrence: changed_fields and single_variable_verified = true. A judge verdict that flips under such a fork is a causally localized defect — the scorer rewarded that one variable, not the work. No static benchmark, and no correlational analysis, can certify that nothing else changed.

The Wind Tunnel is a closed loop over a deployed scorer, running on a tamper-evident evidence spine. The unit of measurement is the exploit; the object of measurement is what happens when you try to fix the scorer and attack it again.

Run the loop for K=5 cycles on a controlled 288-item factorial. Each rubric patch fully closes the exploit it targets (Δ⁺ = 666.7/1k). And yet held-out exploit density — measured each cycle on a still-unpatched signal — does not move: 666.7 → 666.7 → 666.7 → 666.7 → 0, flat until the entire favored-phrase family is manually exhausted.

Patch generalization γ_local = 0.00: a local patch generalizes nothing to the held-out family. That is the operational definition of whack-a-mole, and the preregistered classifier calls it a relocation_plateau. The adversary simply moves to the next unpatched phrase; global exploitability is unchanged cycle after cycle.

An immune system that attacks its own body is worse than the disease. The Wind Tunnel prices every repair with an autoimmunity cost Δ⁻ — the fraction of verified-correct answers the patched scorer newly rejects — and refuses any patch that exceeds a preregistered budget.

One patch, three modalities, same starting scorer:

A second transform family targets the LLM judge’s documented biases rather than the answer extractor. Each keeps the oracle-wrong core — so a judge PASS is a confirmed defect, not a disagreement — and wraps it in natural rhetoric a deployed model could plausibly emit. Flip a card.

The first run reported HumanEval as the most gameable scorer of all: 72.6/1k. A convenient headline — and wrong. The tell was the localizer refusing to attribute anything: zero witnessed single-variable flips on the benchmark with the highest exploit density. Instead of explaining the contradiction away, the authors traced it.

The subtle_bug mutation never matched real solutions and silently fell back to appending a comment (leaving the code correct); meanwhile keyword_inflation prepended prose above Python source, so the file simply stopped parsing. The measured quantity was “prose pasted above code breaks execution and the judge doesn’t notice” — not “the judge accepts subtly buggy code.”

The instrument distills into standing infrastructure: a per-benchmark gameability index — oracle-witnessed exploit density under the canonical adversary, with a bootstrap CI, pinned to the dataset snapshot, judge identity, catalog version, and git SHA. Each index card is content-addressed; anyone recomputes the number offline without trusting the authors.

The ordering is mechanistic, not difficulty-driven: gameability tracks the judge’s ability to independently verify, not the benchmark’s difficulty for models. GSM8K and HLE sit at opposite ends of both scales.

Three instruments, three views, one structure: reference-free scoring is gameable diffusely. Local edits relocate the flaw rather than remove it; even the structural defense that cures the procedural model does not transfer to a real frontier judge. No repair the paper tested removes the exploit surface at deployment. That is the safety-relevant thesis — the alignment problem in miniature: patching a proxy moves the flaw unless you change what the proxy measures.

Disclosure runs as an immune response, not an attack: each broken benchmark’s maintainers receive a content-addressed index card, oracle-witnessed examples, the convergence result for their scorer, and a suggested structural defense — under embargo, before public release. The gameability index ships as a zero-install audit command, a published living index, a CI badge, and an open card schema, so third parties produce and verify cards independently of the authors. And because the preregistration and evidence chain heads are anchored in public timestamp logs, backdating or equivocation by the authors themselves is detectable by anyone.