A final-state benchmark cannot prove the world changed

Most agent benchmarks begin from a frozen state and end with a score. That is enough when the task is a sealed puzzle. It is not enough when the task is work.

Work changes while it is being done. A client revises a priority. A calendar invitation moves. A policy owner retracts an instruction. A new document arrives after the agent has committed to a plan. A manager who can authorize one action cannot authorize another. The model has to notice the change, decide what it means, preserve whatever remains valid, and recover without inventing authority it does not have.

An event-driven environment can stage all of those things. The hard question is whether the benchmark can prove which of them actually happened.

Dynamic is not the novelty claim

Asynchronous events, simulated applications and interruption benchmarks already exist. Treating the broad idea as new would be easy to disprove and would distract from the actual measurement problem.

The narrower claim is more useful: a long-horizon environment needs an evolution witness. It needs evidence that binds a declared occurrence to the state transition the runtime applied, the observation the agent could see, the action boundary at which it became available, and the history a restored sandbox continued from.

Without that witness, a benchmark can be dynamic in its implementation and static in its evidence.

Three green runs that mean different things

Imagine a task in which an agent is preparing a recommendation. Halfway through, an approved source changes one material fact.

In the first run, the event is delivered and the agent adapts. In the second, the event is written to a controller log but never reaches the sandbox. In the third, it reaches the sandbox but not the model-visible observation. All three may end in the same database state. All three may contain a log line saying the event fired. Only one measured adaptation.

The final state cannot separate them. The log cannot separate them either, because the component writing the log is the component whose behavior is in question.

This is the same reason a test should not accept a service saying it wrote a record as evidence that the database contains it. The receipt has to come from the boundary that owns the effect.

The six links in an evolution witness

A useful witness has at least six links.

1. The exact scenario edition and event digest. If any payload, trigger, delivery rule or expected handling changes, the scenario identity changes.
2. A root-only application receipt from the guest runtime. The controller may request an event; the sandbox is the authority on whether the world received it.
3. A materialized world root before and after application. The receipt proves not just that a handler returned success, but which world changed into which next world.
4. The exact projection made visible to the agent. Hidden payload and grader expectations remain hidden; the witness records the hash of the model-visible observation.
5. The logical action boundary and restore generation. Wall-clock timing is provider noise. A boundary such as after the third act or before turn five replays across providers; restore lineage shows which continuation produced later events.
6. A tamper-evident chain head over every occurrence. Dropping, inserting or reordering an event changes the chain.

Each link closes a different substitution. A scenario digest without a receipt proves what was planned, not what ran. A before-and-after root without an observation proves the world changed, not that the agent could know. An observation without a restore generation cannot distinguish a clean continuation from a stale snapshot.

Measure behavior, not hidden thoughts

Once the world evolution is proved, the benchmark can measure visible process behavior.

Did the agent perform the required follow-up? Did it repeat a prohibited stale action? Did it ask for information that the scenario declared requestable? Did it escalate when the available authority was insufficient? How many acts passed before it recovered? Did it still honor an active constraint after three unrelated actions consumed context?

These can be expressed as closed patterns over the transcript and evaluated deterministically. That does not make every judgment deterministic. Some decisions have several defensible outcomes, and some qualities need blinded expert review. The contract should say which constraints are machine-checkable, which outcomes are acceptable, which dimensions require human evidence, and when abstention is permitted.

A single reward should not collapse those distinctions. Task correctness, process compliance, recovery latency and expert judgment are different measurements. If a training service exposes process reward, it should do so under a separately versioned contract rather than quietly changing the meaning of the old one.

A realistic organization is a permissions problem

A simulated inbox with invented messages is useful for testing mechanics. It is not a faithful organization mirror.

The difficult parts of an enterprise application are identities, access control, edits, deletions, threads, notification recipients, timestamps and the fact that two people looking at the same system may legitimately see different worlds. A useful twin preserves those boundaries.

The safe architecture separates capture from execution. A narrowly scoped connector performs an authorized read. Credentials terminate at that boundary and never enter the frozen world. The resulting records are minimized through an approved field allowlist, mapped to stable benchmark identities, stamped with their original visibility projection, and frozen with source, ACL, transformation and record-set digests.

The runtime is read-only. An agent can search and read what its declared identity is allowed to see. A write back to the real organization is structurally unavailable. A scenario can inject a new frozen record into the sandboxed twin, but that changes only the replay world.

Rights are independent of access. Permission to read a channel does not imply permission to retain it, use it for evaluation, turn it into training data or publish it. Those are separate approvals and should remain separate fields. A training export must refuse a capture authorized only for evaluation.

Live read-only access has a place during exploration, but it cannot make a reproducibility claim. The source may change between runs. A certified benchmark freezes the minimized capture and replays it.

The interruption needs a control arm

One interrupted run explains almost nothing. The agent might have failed the base task anyway.

A defensible study predeclares a family of paired conditions over the same tasks and seeds: a no-event baseline, the interruption, a one-axis counterfactual where useful, and sometimes a distractor that consumes attention without changing the task state. Every comparison pairs only identical task-and-seed cells. If either arm suffers an infrastructure failure, that pair leaves the denominator. Cells present in only one arm are reported, not silently dropped.

That produces several readings instead of one flattering delta: baseline success, interrupted success, paired task lift, process lift, recovery acts, stale-action count, and delayed constraint retention. Confidence intervals belong on the within-pair differences. Below the declared sample floor, the interval may be shown but no significance claim is made.

The experiment can then answer a real question: did the event expose a capability gap, or did the harness merely make the task harder in an uncontrolled way?

What this evidence still cannot prove

An evolution witness does not prove that a scenario is representative of a customer's work. Rights receipts do not prove that the chosen source scope was the right one. A deterministic process pattern does not settle an open-ended professional judgment. Provider parity is not established until the same frozen scenario produces the same visible semantics and evidence shape on each owned runtime.

Those are not footnotes. They are the next gates.

The research question worth running is whether this witness detects defects that terminal-state checks and ordinary controller logs miss. Build paired forks, mutate one link at a time, and measure incremental detection. Duplicate an event after restore. Change the world without an occurrence. Deliver a projection to the wrong identity. Reorder two events. Widen a capture scope. Drop a deletion. Reflect a secret-shaped field. If the proposed witness does not catch more of these than the baselines, the mechanism is paperwork, not progress.

The one-line version

A frontier environment must prove not only the agent's final answer, but the world evolution that answer was made inside: exact event, root receipt, before-and-after state, visible projection, action boundary, restore lineage and a signed chain that survives offline checking.