When the simulated user leaks

A large fraction of the work you actually want an agent to do begins with an incomplete request.

"Can you look into the Henderson claim?" A capable colleague asks which policy year, notices the claim was reopened, and asks whether the earlier settlement is in scope. An incapable one guesses, and produces a confident answer about the wrong thing.

That difference is most of what separates a useful agent from a demo, and you cannot measure it with a static prompt. The task has to contain a person who knows things and doesn't volunteer them.

Why the scripted version stops working immediately

The first implementation everyone builds — including us — is a rule table. If the agent's message contains certain keywords, reveal the corresponding fact. Otherwise reply with something neutral.

It works exactly once.

An agent under test learns within a turn or two that the counterparty is a keyword trigger, and from then on the optimal strategy is keyword enumeration: list every plausible term, harvest whatever comes back. You are no longer measuring whether the model knows what to ask. You are measuring whether it can brute-force a lookup table, which nothing in production rewards.

The neutral filler makes it worse. A persona that answers every unmatched message with the same acknowledgement is trivially distinguishable from a person, and a model that detects it will behave differently than it does with a real user — which is the one thing an evaluation must not permit.

So you reach for an LLM to play the person. Correct instinct. The obvious implementation is a serious mistake.

The mistake: asking a model to keep a secret

The natural design is: give the persona model everything — the full brief, all the facts, and an instruction saying which ones to withhold until the agent asks properly.

You have now placed the answer key inside a prompt that the system under test is allowed to send arbitrary text to.

Every failure mode follows from that one sentence:

• The agent asks a leading question and the persona helpfully confirms.
• The agent says "just summarise everything you know" and a compliant model complies.
• The persona simply forgets the instruction under a long conversation, which models do.
• The agent injects a plausible-looking system instruction and the persona follows it.
• The persona hints — mentions the shape of what it's holding back — and a strong model infers the rest.

And the failure is silent. Nothing throws. You get a transcript that looks fine, a task that scores fine, and a difficulty mechanism that has quietly stopped existing. Worse: it fails preferentially on the strongest models, because the better the agent is at eliciting information, the better it is at eliciting it improperly. Your hardest tasks get easiest for the models you most need to distinguish.

The fix: withhold outside the model

The load-bearing decision is a single structural one.

A fact whose disclosure condition has not been met is never placed in the persona's prompt at all.

The gating stays where it can be enforced: deterministic evaluation of the disclosure condition, in code, before any model call. The model receives the persona description, the conversation so far, and only the facts already earned.

It cannot leak what it was never told. Not "is instructed not to leak" — cannot, in the way a program cannot print a variable that was never assigned.

The model's remaining job is narrow and safe: phrase the available material in character. That's the part LLMs are genuinely good at and the part where failure is cosmetic.

This inverts the usual intuition — you might expect the smarter component to handle the sensitive decision. Exactly backwards. The deterministic component makes the security decision; the probabilistic component handles the prose. Any design where a model's compliance is the only thing standing between an agent and the answer key is a design that fails open.

Check that it worked

Structure prevents leaks through the prompt. It doesn't prevent a model from inventing a fact that happens to be true, or from paraphrasing its way toward one.

So check the transcript afterwards: every gated fact appearing in a persona turn must have been earned before that turn. It's a string comparison against the conversation, not a judgement call. Deliberately so — using a model to grade whether a model kept a secret from a third model is a tower you do not want to be standing on.

The subtle part is what a detected leak should do.

Not fail the agent. A persona that leaked is your bug, not the model's, and scoring the agent down for it would be punishing a candidate because the invigilator read the answers aloud. It would also make your benchmark quietly easier and noisier at the same time.

Instead, void the episode. Mark it an environment failure and remove it from the denominator entirely — not counted as a pass, not counted as a fail, excluded. A simulator that leaked cannot make the model look better or worse; the cell is simply blank, and the blank is reported.

That's a general rule worth adopting well beyond this: when your infrastructure fails, exclude the result, don't score it. A pass rate that silently includes cells your own harness broke is a pass rate about your harness.

The reproducibility problem, and its answer

Putting a model in the environment breaks the property that makes a benchmark a benchmark: run it twice with the same seed, get the same result.

The answer is to treat the persona's contribution as data, not as a live component. Pin temperature to zero and seed per episode; record the persona's model identifier and parameters alongside the run; and when checking reproducibility, compare against the recorded persona transcript rather than re-sampling it.

The world stays a pure function of its inputs. The persona's output becomes part of the recorded environment, exactly like a captured web page or a frozen database.

And if the run's manifest doesn't say which persona model produced those turns, that run is not reproducible — and it should say so, loudly, rather than pass a check that no longer means anything.

What we're deliberately not doing

No model judging whether the persona "felt realistic." Unfalsifiable, and it puts a model in the reward path, which is the thing the rest of the architecture exists to prevent.

No persona that can take actions. A simulated user that answers questions is an environment. One that can change the world is a second agent, and no seed pins the interaction between two of them.

The one-line version

If your simulated user is holding facts it's been told to withhold, the answer key is in a prompt your system under test can write to — and the models best at extracting it are exactly the ones you're trying to measure.

Withhold outside the model. Then check the transcript. Then void the cell when it leaks.