Reward Hacking Loop

01 · The proxy problem

Goodhart's law, running at gradient speed

When a measure becomes a target, it stops being a good measure. RLHF and RL-with-verifiable-rewards make the measure a target millions of times per run: the optimizer queries the reward precisely to find its highest-scoring behaviours. If the proxy and the true objective ever diverge, optimization pressure will find the divergence and live in it.

Prior work mostly measures the wreckage from outside — over-optimization scaling laws, catalogues of specification gaming, formal definitions over policy pairs. Essential, but three operational questions remain unanswered inside a running loop:

When does hacking begin? Which feature of the reward is being exploited? And does patching that feature cure the problem — or merely relocate it?

The paper's answer is a single instrument, carried in from the earlier Reward-Hacking Wind Tunnel (Pillar 3): a machine oracle that occasionally taps the loop and witnesses, with no human label, actions the reward graded correct that are provably wrong.

The instrumented loop. The optimizer only ever sees the reward channel. The oracle sits outside it — too costly to use as the training signal on every step, but cheap enough to spot-audit. A witnessed exploit is an action the reward passes that the oracle proves wrong: PASS(a) ∧ ¬ oracle(a).

02 · A minimal arena

Three moves, one temptation

To watch the mechanism without the fog of a full language model, the paper builds the smallest setting where reward hacking is a genuine option the optimizer can choose — not an artefact imposed on it. A single-step task; a policy picking among three actions.

Two reward channels score them. VerifiableReward returns the oracle exactly — ungameable by construction. GameableReward also passes anything wearing a judge-favoured cue — a scorer failure mode borrowed from Pillar 3, reused as a training reward. Same task, same optimizer, same seed; only the reward's exploitability differs.

SOLVE

Attempt the task honestly. Correct with probability p_solve — the model is imperfect.

oracle: sometimes ✓reward: sometimes

EXPLOIT

Always wrong — but wears the cue the gameable reward loves.

oracle: never ✓gameable: always PASS

HEDGE

Plain wrong, no cue. The boring control that separates "hacking" from "just failing".

oracle: never ✓reward: never

Verifiable reward — P(EXPLOIT) after training

0.01

Gameable reward — P(EXPLOIT) after training

0.98

The choice, made by gradient descent. Under the verifiable reward the policy learns to SOLVE. Give it a reward with a seam, and the same optimizer pours 98% of its probability mass into EXPLOIT. Nothing else changed.

03 · Emergence

The Goodhart gap opens

Plot proxy reward and true (oracle) quality on the same axes and the failure is not subtle. Against the gameable reward, the proxy climbs to 0.98 while true quality falls to 0.00 — a gap of +0.98, with the two curves anti-correlated at −0.84. The reward's dashboard says the run is going beautifully, precisely while the task is being abandoned.

The matched control run — identical in every way except that its reward equals the oracle — shows proxy and true tracking exactly, both settling at 0.62 with zero gap. The gap is caused by the reward's exploitability, not by RL itself.

proxy reward true quality (oracle) Goodhart gap

Verifiable reward control

Gameable reward hacked

Same optimizer, same seed, same task. Left: reward = oracle, so proxy and truth are one line (gap 0.00). Right: the reward has a seam, and the policy climbs into it — proxy 0.98, true 0.00, correlation −0.84. A terminal proxy alone would report the opposite of what happened.

04 · Localization

Caught in the act: the witnessed fork

Knowing hacking happened is not knowing what is being exploited. The localizer is an exact single-variable intervention: take each oracle-witnessed exploit a_i, apply a transform T_x that removes exactly one candidate reward feature x — preserving the oracle label and every other recorded field — and watch whether the reward's verdict flips.

φ_x = witnessed flip rate, β_x = negative-control flip rate, Δ_x = φ_x − β_x

The same transform is applied to oracle-correct, reward-passing answers as a negative control: a transform that indiscriminately breaks passing answers proves nothing. The feature is validated on the witnessed sample iff φ_x > β_x and φ_x > 0.

witnessed exploit

"…and therefore the answer is 41." (wrong)

reward · PASS oracle · ✘

negative control

"…so the answer is 28." (correct)

reward · PASS oracle · ✓

Watching 64 witnessed exploits…

0.00φ — flip rate on exploits (64/64) 0.00β — flip rate on controls (0/64) 0.00Δ — localization lift, n = 64

One variable, forked. Removing the cue flips every witnessed exploit from PASS to FAIL (φ = 1.00) and flips none of the 64 correct controls (β = 0.00): lift +1.00. Under the stated intervention assumptions — T_x changes x alone — the cue is validated as the exploited feature on this sample. It is a feature-, channel-, and sample-specific claim, not automatic causal discovery.

05 · Repair

Whack-a-mole, then a cure

Localization hands you a patch target. But a reward fooled by several cues raises the practical question every red team knows: does fixing the exploited cue fix the problem, or does the policy simply migrate to the next seam?

The paper runs the experiment with a reward fooled by three cues. Patch the localized cue, retrain, re-measure — and watch the exploit relocate. Only when every cue is patched does the gap close. The whole sequence is summarized by a relocation share:

γ_local = #distinct cues exploited − 1#patches applied = 3 − 13 ≈ 0.67

γ_local = 0 would be a clean cure on the first patch; 0.67 means the repair was mostly whack-a-mole before the eventual cure.

cue A

idle

cue B

idle

cue C

idle

Goodhart gap after retraining0.00

γ_local = (3 − 1) / 3 = 0.67 — two relocations across three patches: whack-a-mole before the cure.

Patch, retrain, relocate. Patching cue A sends the policy to cue B; patching B sends it to C; only the comprehensive patch closes the gap. (This share runs opposite to the Wind Tunnel's patch-generalization γ and is measured on a different object — the two are not numerically comparable.)

06 · Breadth

No safe objective, no innocent reward

Is this a quirk of policy gradients? No. DPO trains no explicit reward model at all — it fits the policy directly to preferences — yet its implicit reward learns the same exploit from gameable preference data. Removing the reward-model training stage is not, by itself, a safeguard.

Policy gradient — P(EXPLOIT)

gameable reward

0.98

verifiable control

0.01

DPO (implicit reward) — P(EXPLOIT)

gameable preferences

0.66

verifiable control

0.13

Dose–response in pressure. Sweeping KL-regularization temperature, P(exploit) climbs 0.13 → 1.00 while true reward falls 0.61 → 0.26 — optimization pressure converts directly into exploitation (95% bootstrap CIs).

Learned rewards inherit it. A logistic reward model trained on data where the cue spuriously correlates with correctness learns to weight the cue (0.0 → 5.1); the policy exploits it, and the witnessed fork recovers the learned feature. The pathology is learned, not hardcoded.

Same seam, four faces. The mechanism spans the implemented policy-gradient loop and DPO's implicit reward, scales with optimization pressure, and emerges in learned reward models as a dose-response in training-data bias.

07 · Detection

A training-time immune system

Everything so far is diagnosis after the disease. The payoff is turning it into an early-warning signal: each training window, draw a seeded audit sample from the current policy, run the oracle on it, and estimate the Goodhart gap and witnessed-exploit rate. The detector alarms only when both exceed fixed thresholds for k consecutive windows — persistence, so one noisy audit can't cry wolf.

audit size 96 gap threshold 0.50 exploit threshold 0.45 persistence k = 3

Fires before saturation. On the gameable reward the alarm fires at iteration 12 with the gap at 0.60 — well before it saturates at 0.98. The matched verifiable-control trajectory (flat green) raises zero alarms. One control trajectory establishes expected behaviour for this seed, not a population false-positive rate — that needs repeated seeds. Cost: one oracle call per sampled action per window.

08 · Out of the toy

A real policy, a frozen bet

Does reward-channel separation survive contact with a neural policy? The paper runs a frozen, one-seed, paired GRPO diagnostic: Qwen2.5-0.5B-Instruct with LoRA on GSM8K math problems — 128 training prompts, 64 held-out, seed 20260901, 300 optimizer steps, everything matched between arms except the reward. The control rewards exact final-answer correctness. The gameable arm rewards correctness or the mere presence of the word "therefore".

The prediction was frozen in advance: the gameable arm's final proxy–oracle gap should exceed 0.10; the control's should stay at zero.

proxy ✓ and oracle ✓ wrong but rewarded fails both

Exact-match control · 64 held-out items

proxy 13/64 · oracle 13/64
gap 0/64 — at all 13 evaluations

Gameable reward · 64 held-out items

proxy 58/64 · oracle 1/64
wrong-but-rewarded 57/64

final Goodhart gap 0.890625 — exceeds the frozen 0.10 threshold; H1 supported under the registered rule.

The reward got easier while the task got worse. The gameable arm's baseline already held a 6/64 seam of wrong-but-rewarded completions; optimization amplified it to 57/64 (first crossing 0.10 at step 25, above it at all 12 post-baseline evaluations; 994 wrong-but-rewarded training samples vs. zero in the control). Both arms lost oracle accuracy — no capability claim is made. Both final adapters reproduce their complete 64-row evaluation digests under fresh model-backed replay.

09 · Scope & takeaways

What this buys, and what it doesn't

An oracle-witnessed intervention instrument buys three things a scalar reward curve cannot: it says when hacking starts, which tested feature is implicated under an exact fork, and whether a fix is a fix. Detection plus repair is a training-time immune system — the defensive counterpart to the offensive Wind Tunnel. And the real-policy pair delivers the uncomfortable practical lesson: a reward can become dramatically easier to satisfy while the underlying task collapses, and a terminal proxy alone would report the opposite of what happened.

What the paper shows

A precise in-loop exploit definition — PASS(a) ∧ ¬ oracle(a) — with witnessed, negative-controlled localization (lift +1.00, n = 64).
Repair quantified as cure vs. relocation (γ_local = 0.67 — whack-a-mole before cure).
The mechanism spans policy-gradient and DPO paths, rule-based and learned rewards.
An online detector that fires before saturation, with zero alarms on the verifiable control.
A reproducible real-policy diagnostic: gap 0.890625 vs. 0, gated by 55 property/control checks, with exact model-backed replay.

What it deliberately does not

The minimal model isolates the mechanism; it does not estimate real-world magnitudes.
One 0.5B model, one seed, one deliberately exposed cue — no variance estimate, no claim about frontier models or production reward models.
The gameable baseline already contained the seam (6/64): this is amplification of an existing seam, not emergence from zero.
The localizer needs a candidate variable and exact intervention fidelity; entangled features can violate that.
One control trajectory ≠ a false-positive rate; real-policy localization and repair (M4–M5) are future work.