A failure map is not a training plan

A good evaluation can identify a repeated failure with extraordinary precision.

It can show the exact task, the observation available to the agent, the first bad act, the downstream consequence, the criterion that failed, the evaluator that issued the verdict and the replay that reproduced it.

That is a failure map. It is not yet a training plan.

The distance between the two is where many evaluation programs quietly lose scientific validity. They select examples from a failing test, fine-tune on them, test on nearby examples, observe a higher score and call the model improved. The intervention, measurement and selection process have become one loop. The answer key is now part of the lesson.

Start with a claim, not a dataset

A training study should name the behavior it intends to change before generating examples.

For example:

When approved evidence changes after planning begins, the candidate model will re-read the affected source, update the dependent decision and avoid using the retracted value, without regressing on unchanged cases.

That claim identifies:

• the causal slice to target;
• the observable behavior that counts as improvement;
• the control population that must remain stable;
• the held-out challenge that tests generalization; and
• the failure that would falsify the intervention.

“Improve the benchmark score” does none of those things.

Rights are part of experimental validity

An organization may have permission to evaluate a trajectory and no permission to train on it. It may be allowed to retain a derived label and prohibited from retaining the raw customer content. It may be allowed to publish aggregate statistics and prohibited from redistributing task artifacts.

Those are separate rights:

• evaluation;
• internal retention;
• derived publication;
• redistribution;
• training use; and
• raw-trajectory use.

The training compiler should refuse any example whose artifact-level rights do not include the intended use. A project-level checkbox is too coarse. One task can combine a customer document, a licensed policy, a synthetic message and a human review. The narrowest relevant right governs each export.

This is not only compliance hygiene. If the provenance of the training set cannot be reconstructed, neither can the experiment.

Split by content identity, not row number

Random row splits are inadequate when examples share a source document, template, scenario seed, case family or generated parent.

Near-duplicates can cross from training into validation while carrying different IDs. A model then appears to generalize because it recognizes the structure it was just taught.

A safer split binds each artifact to stable content and lineage digests, then groups related material before assignment. The training, validation and held-out sets must have disjoint content identities and disjoint declared parent families where the study requires them.

Check the split after every generation or filtering step. A clean source split can become contaminated when a synthesis pipeline produces variants from both sides and merges them under new IDs.

Compare controlled cells

The baseline and candidate must face the same held-out cells:

• task edition;
• world state and event schedule;
• runtime and scaffold;
• seed;
• tool and network policy;
• attempt budget;
• evaluator edition; and
• exclusion policy.

Only the declared intervention should differ.

If the candidate gets a better prompt, a longer timeout and a repaired runtime, the observed change is a bundle. It may be operationally useful, but it does not isolate training lift.

Keep the baseline record even when it is embarrassing. A candidate score without its paired baseline cannot distinguish capability improvement from an easier edition.

Measure variance at the item level

Agent outcomes are noisy. Multiple attempts on the same task are correlated because they share the same source, structure and evaluator.

Report seeds and attempt policy, but do not treat every attempt as an independent task. Uncertainty should be clustered by the unit sampled from the target population, usually the task or case family.

At minimum, preserve:

• paired outcome deltas;
• task-clustered intervals;
• per-criterion changes;
• regression counts;
• infrastructure exclusions; and
• the number of items, seeds and physical attempts behind each estimate.

A higher mean with a wide interval is a hypothesis for the next study, not a release claim.

Test the intervention against evaluator shortcuts

Training can improve a score by teaching the candidate how the evaluator talks.

Probe for:

• reference phrasing copied without grounded content;
• rubric keywords inserted without the required action;
• excessive verbosity that exploits judge preference;
• format compliance replacing substantive completeness;
• memorized benchmark identifiers;
• confidence increasing while calibration worsens; and
• a scorer gain that disappears under a semantically equivalent presentation.

This is where the evaluator wind tunnel and the training study meet. The candidate should face patched evaluators, held-out attack families, meaning-preserving transformations and an independent validity measure. A gain that survives only the evaluator used during development is not robust lift.

Preserve the failures that did not improve

Training reports often focus on the average and hide the trade.

A useful comparison separates:

• fixed failures;
• unchanged failures;
• new regressions;
• apparent gains caused by exclusions;
• performance under distribution shift;
• calibration changes; and
• suspected reward-hacking behavior.

Slice the result by task family, difficulty, failure mode, interruption phase and required skill. A model can improve overall while becoming worse on the small set of high-severity cases the study was created to protect.

Pairwise preference is useful, but it answers a different question from criterion scoring. A reviewer may prefer the candidate's response while both responses miss a mandatory fact. Keep preference, correctness and process compliance as separate views.

Compile a steering order before dispatch

The handoff from evaluation to training should be a versioned object, not a folder of examples.

A steering order should bind:

• source proof pack and manifest;
• artifact rights;
• split and lineage digests;
• failure families and causal slice;
• proposed intervention;
• baseline and candidate cells;
• seeds and uncertainty method;
• spend and time limits;
• reward-hacking and autoimmunity probes;
• hidden-vault policy;
• stop conditions; and
• the exact claim boundary.

Compilation can verify that the design is coherent. It cannot authorize training by itself. Dispatch remains a separate, authenticated decision with spend, data and organizational approval.

The honest status is prospective

Before the controlled study runs, the strongest eligible statement is:

Verified failures produced a training-steering proposal that passed rights, split-integrity and design checks. Training dispatch and realized lift remain unmeasured.

That sentence may sound conservative. It is also the line that keeps an evaluation company from becoming a machine that launders interesting failure analysis into unsupported improvement claims.

Once the study runs, the evidence can support more: exact paired lift, regressions, variance, generalization, alignment with human review and whether the evaluator itself remained stable.

The failure map tells you where to look. The controlled steering order earns the right to intervene. Only the completed, replayable comparison earns the right to say the model got better.