Interruptible Universes

The right answer is not the whole story

Start with one trace from the paper's released reference world. An agent works an authority-conflict episode and lands on the correct terminal recommendation — but it cites an unverified deal-team chat message as evidence. Terminal accuracy says correct. The evidence-aware judge says 0.8, because the provenance failure is real: outcome agreement does not make a source authoritative.

Now go one level deeper. Suppose the environment declares that a policy was revised after the agent's second act. The final database contains the revised policy. The log contains a revision event. Everything looks fine. Yet at least three different histories are compatible with that same ending — and only one of them is the episode the benchmark promised.

The stakes run in both directions. If an invalid environment episode is scored as an agent failure, a leaderboard quietly measures harness defects. If it is admitted into a training corpus, the reward signal can favor stale evidence, unauthorized instructions, or behaviors that exploit replay bugs. But an evidence system that rejects valid episodes too eagerly makes long-horizon evaluation impractical. The paper's empirical question is exactly that trade: does a particular evidence composition catch more invalid episodes without unacceptable false rejection or cost?

Four ways an episode goes wrong — separated before scoring

Terminal correctness conflates four distinct fault classes. The witness's job is to pull them apart: did the environment actually apply the declared change? Was the change disclosed to the agent exactly as declared? Did the notice carry legitimate authority? And only then — did the agent adapt? The first three are instrument faults. They get an episode excluded and attributed, never counted against the model.

Diagnosis stays observable: if contract and roots disagree, blame the environment; if roots are right but the projection differs, blame disclosure; if both are right and the agent submits stale state, the failure is attributable to actions — without pretending to know the model's internal reasoning.

Anatomy of an evolution witness

For every declared exogenous occurrence, the benchmark emits one canonical receipt. Six fields, canonically encoded and hashed together (SHA-256 over canonical JSON — a content digest, deliberately keyless, not a signature). Change any one field without coherently changing the trusted source object, and some equality in the verifier breaks.

The paper is equally explicit about what a witness does not prove. It tests internal consistency under a stated trust boundary — trusted canonicalizer, root-owned applicator, honest fixture authoring. It cannot detect a root owner who lies consistently across contract, roots, and receipts; it does not prove an external fact was true, that the model read or understood a notice, or that the host was uncompromised. Study A even includes defects the witness is expected to miss, so coverage is falsifiable rather than 100% by construction.

A universe small enough to audit end to end

The released artifact is a fully synthetic, deterministic reference world — an underwriting-shaped fiction (explicitly not lending guidance) chosen because its state changes are easy to inspect. The agent reads a case and a policy, may check an inbox, computes a debt-to-income metric, and submits one of APPROVE, CONDITIONAL, DENY, ESCALATE. Case + policy form the material root; the inbox is deliberately excluded, so an informational notice never has to masquerade as a material mutation.

Every declared event becomes eligible after act 2, and each of the five scenario editions exercises one validity property:

What is validated today — and only this

Everything measured so far is deterministic instrument behavior. Three scripted control policies — program logic, not models, not personas — exercise known judge branches: interrupt_safe re-reads authoritative state after an interruption; stale_context sees the notice but commits from cached state; message_credulous treats a recommendation-shaped message as sufficient evidence.

Two committed audit traces show the separation working. In policy-revision × stale_context, the policy submits APPROVE against a superseded rulebook: score 0.2, labeled stale state, missed change, decision error, evidence gap. In authority-conflict × message_credulous, the answer is correct but cites the unverified message: score 0.8, authority_violation — the motivating trace from the top of this page.

The release also ships a deliberately non-confirmatory engineering preflight: 26 isolated synthetic forks of valid parents, each projected through five evidence views. The question is narrow — does the paired edit even remain visible in that projection? An upper bound on information available to a detector, nothing more.

What Study A will test: does the witness actually catch more?

Study A is the paper's primary scientific test, and none of it has been run. From frozen valid episodes it will create paired invalid forks — one isolated mutation each, unrelated bytes preserved — and ask five evidence policies to classify each episode and localize the first faulty origin. The mutation families are grounded in documented benchmark and runtime failure modes, not reverse-engineered from witness fields:

The comparators matter as much as the mutants. An ordinary event log is not the strongest plausible baseline, so the design mandates a proof-of-execution-style causal comparator, with a published field mapping and conformance suite before any confirmatory run:

The primary endpoint is a paired detection contrast (exact McNemar on discordant pairs) between W and the strongest baseline; false rejection must stay within a 5% noninferiority margin; at least 60 independent faithful parents anchor the corpus, with margins, sample size, and stopping rules frozen before any outcome is inspected. Then come the ablations — remove one witness field at a time:

What Study B will test: do live agents actually adapt?

Study A tests evidence; Study B asks the behavioral question — and only inside episodes that pass environment-integrity checks first, so an invalid episode can never be misread as model capability. Identity-bound attempts (exact model ids, prompts, scaffolds, budgets frozen before execution; a recorded seed is an attempt, never proof of determinism) face the same admitted world change under two experimental factors: where in the trajectory it lands, and how it is delivered.

A second frozen axis holds the boundary fixed (after metric calculation) and varies who says so — the authority ladder. Only the top rung changes material state; every rung still gets a full witness, because authority failures are precisely the ones that leave the material world untouched:

1. A1Root-owned binding sourcethe environment itself revises the policymaterial change
2. A2Delegated verification channelauthorized, but indirectnotice only
3. A3Authenticated human outside policy scopereal person, no authority herenotice only
4. A4Unverified internal messagethe deal-team chat from the motivating tracenotice only
5. A5Unsupported external instructionshould never enter the evidence setnotice only

Ten candidate/control fixture pairs for these axes are already frozen and pass 100/100 isolated construction probes — a corpus-integrity result only. Whether any phase or rung actually changes live-model behavior is exactly what remains unmeasured.

The claim ladder: what may be said, and when

The draft carries an authoritative status registry (its Appendix A) and a claim ledger mapping every candidate claim to the evidence it would require. Three strata, kept visibly apart:

Advancement from protocol to empirical paper is gated, and the gates are decision rules, not vibes:

1. Preregistration predates any confirmatory outcome inspection
2. Baseline implementations pass their own conformance tests
3. Mutation forks independently audited for isolation
4. Every reported denominator reconciles to receipt status
5. Faithful false rejection acceptable under the frozen rule
6. Human and model outputs have documented rights
7. Claims follow the locked claim-to-metric map
8. Exact released files pass deterministic verification

What this composes with — and what it doesn't claim

The paper is unusually careful about its non-contributions. Dynamic worlds and interruptions come from Gaia2/ARE and InterruptBench; stateful tool evaluation from τ-bench, AppWorld, ToolSandbox, OSWorld; provenance from W3C PROV; execution evidence from Proof of Execution and CAVA; branching substrates from Agentic Data Environments and BranchBench; hash chains from 1991. None of that originates here — and the paper says so in a table:

The design philosophy underneath is worth stating plainly: the evolution of the world is treated as part of the benchmark's measurement instrument. A scenario doesn't merely declare an interruption, and a runner doesn't merely log one — the evidence chain must connect declaration to state transition, disclosed observation, action boundary, restore history, and judgment. The cost is more evidence and more trusted code; the planned ablations exist to find the smallest witness that still holds the chain together.

Until the sealed result editions land, this remains what it says on the stamp: a precise, auditable, deliberately falsifiable proposal — with its empty tables published in advance.