Your traces are a benchmark you haven't compiled

If you have an agent in production, you are already sitting on the most valuable evaluation asset you will ever own: a log of real users asking for real things, and the system's attempts at them.

Nobody made those up. Nobody sanitised them into the shape that makes a demo work. They are the actual distribution of the actual job.

Converting them into an evaluation set is a well-defined engineering problem, and the interesting thing about it is exactly where it stops being mechanical.

What a trace gives you for free

A production trace hands over most of a task without any work:

The request. The user's own words, in the register users actually use — which is a long way from how a prompt engineer writes an example.

The context. What the system could see at the time: the documents, the account state, the tool responses. That's your world fixture, already assembled, already realistic.

The tool sequence. Which capabilities got used, in what order. That's your action interface and your milestone structure, derived rather than guessed.

The difficulty distribution. Real traffic is mostly easy with a heavy tail. A hand-written set is mostly medium, because that's what feels worth writing down. The shape of real work is itself information.

The frequency. How often each request pattern occurs, which tells you what to weight.

That's five of the six things a task needs, and none of them required a judgement call.

What a trace does not give you

The sixth thing is the answer key. A trace records what the system did. It does not record what the system should have done.

This is the entire problem, and it's where naive trace-to-eval pipelines quietly fail. If you take the agent's own output as the expected answer, you have built a benchmark that measures agreement with the past — a regression test for behaviour, dressed as a test of correctness. Every mistake in production becomes a mistake the benchmark now requires.

There are only a few honest ways to get a key:

Ground it in the world, not the transcript. If the request was "what's the coverage limit on this policy," the answer isn't whatever the agent said — it's what's in the policy record. Read it from the fixture. This is the best case and it's more common than people expect, because a large fraction of real requests are lookups against state you have.

Ground it in what happened next. Sometimes the outcome is recorded downstream: the ticket was resolved this way, the transaction settled at that figure, the human corrected it to this. That's a genuine key from a different source.

Have a human supply it. Expensive, reliable, and the only option for genuinely judgement-shaped requests. Reserve it for the traces worth the money.

Refuse. If none of the above apply, the trace does not become a task. It becomes a documented refusal that says why — "no verifiable answer available for a judgement-type request" — so the person reviewing the pipeline sees the reason instead of a mysterious gap.

That last option is the one that distinguishes a real pipeline from a demo. Most traces cannot become tasks. A pipeline that converts 100% of its input is not thorough; it's fabricating keys.

The failure mode to watch for

Here is the specific way this goes wrong, and it is subtle enough that we hit several variants of it.

Your compiler decides a trace has everything it needs and marks it complete. It goes into the set. And then the next stage — the one that actually validates tasks — rejects it, every single time, for a reason the "complete" label never considered.

We shipped exactly this: tasks confidently labelled fully-grounded that failed their own correctness probe 100% of the time, because the compiler had grounded an answer key that the task's checker structurally could not read. In another variant the key's type was wrong — a number handed to a checker that compares sets. Neither produced a warning. Both produced a complete-looking task.

The rule that prevents it: "complete" must mean "survives every downstream check," and that has to be verified rather than asserted. If your compiler can emit a status, something must confirm that status against the gates it claims to have satisfied. Otherwise the label drifts from the gate, and the drift is invisible until someone asks why the set is smaller than the count.

Privacy is a design constraint, not a step

Production traces contain customer data. Some of this is obvious and some isn't.

• Names, account numbers, and free text are the easy part, and standard redaction handles most of it.
• The context documents are the harder part, because they're the fixture and redacting them can destroy the task. A policy document with every figure masked no longer supports the question.
• Uniqueness is itself identifying. A trace can be scrubbed of every field and still be the only one of its shape, which matters a great deal if the eval set leaves your perimeter.

Two structural answers, in order of preference: keep the compilation inside the customer's environment so raw traces never move, and shift from redaction to substitution — replace the real entities with generated ones consistently across the fixture, preserving the structure and the difficulty while changing every value.

Decide which of those you're doing before you compile anything, because retrofitting it means recompiling everything.

Provenance, and why it needs its own label

A trace-derived task is not the same object as an authored one, and it should never be indistinguishable from one downstream.

It has different failure modes (a key grounded from a fixture that later changes), different rights (the customer's data underlies it), different review needs (a human should see the first N of every new pattern), and a different lifetime (traffic drifts, and a task derived from last year's traffic may describe work nobody does).

Give it its own provenance value from the start. Retrofitting provenance is one of those migrations that's technically easy and organisationally impossible, because by the time you want it, nobody can tell which rows were which.

What to actually do

1. Take 100 traces, sampled to match your traffic mix rather than the interesting ones.
2. Cluster them by request pattern. You'll usually find that 100 traces are 12 shapes.
3. For each shape, ask where the key would come from. Fixture state, downstream outcome, human, or nowhere.
4. Compile only the shapes with an honest key. Count the refusals — that number is a finding, not a failure.
5. Run the compiled tasks through your normal validation before believing any label your compiler produced.

You will end up with fewer tasks than traces, and they will be worth more than anything you would have written from scratch, because you did not choose them.

The one-line version

Your traces already contain the questions. They do not contain the answers — and every trace-to-benchmark pipeline that doesn't refuse the traces it can't ground is quietly certifying that yesterday's behaviour was correct.