How do you know the gate would catch it? Seed the corruption

Every serious evaluation system accumulates gates. Checks that run before a dataset is accepted, before a result is published, before a model is promoted. They fire occasionally, somebody fixes something, and over time the team develops a comfortable sense that the corpus is in good shape.

Ask what the detection rate of any individual gate is, and the room goes quiet.

This is not an embarrassing gap in an otherwise rigorous practice. It is the central one. A gate you have never measured is indistinguishable from a gate that does nothing, and the two produce exactly the same observable in normal operation: silence. Your corpus is clean and your gate is working, or your corpus is filthy and your gate is blind. From the inside these look the same.

The experiment is available and nobody runs it

The measurement is not hard. Take a corpus you believe is clean, inject defects of a known kind at a known density, run the gate, and count what came back.

That is the whole design. It is the standard method in every field that takes detection seriously — spiked samples in an analytical lab, seeded faults in hardware test, known-positive controls in a screening assay. Software evaluation is one of the few places where a detector routinely ships with no sensitivity figure at all.

The reason it does not get run is not difficulty. It is that running it produces a number, and the number might be low, and once you have a low number you own it.

One corruptor per gate, and none of them clever

The first design decision is what a defect is, and the temptation is to be creative.

Resist it. Each of our gates has exactly one corruptor paired to it, and each corruptor injects the narrowest possible instance of what that gate claims to catch: a citation pointing at a document the corpus cannot read, a pair of rules that contradict each other, a rule whose effective date sits past the declared horizon, a supersession pointing at a rule that does not exist, an assertion with nothing attesting it.

The corruptor also declares, in advance, which blocker the gate is expected to emit. That declaration is what makes the result interpretable rather than merely encouraging. A run in which the gate emitted some complaint is not a run in which the gate caught the thing you injected — that distinction is the difference between a detection rate and a coincidence rate.

Two temptations worth naming explicitly, because both were live for us:

Do not generate corruptions with a language model. It is a natural idea and it destroys the experiment. You lose the guarantee that a defect was injected at all, you lose the guarantee it is the defect you meant, and you can no longer say what density you achieved. A detection curve is a measurement against a known truth; the moment the truth is itself a sample from a model, you are measuring two things and can separate neither.

Do not inject something too easy to be interesting or too obscure to be real. The corruptor should be the plainest instance a competent adversary would produce by accident. If the gate cannot catch the plain case, its performance on the clever case is not a question anyone needs to ask.

The control arm is not optional

Before any detection figure is produced, we run the same pipeline with a corruption density of zero.

If the gate fires anything on the uncorrupted corpus, the entire measurement is withheld. Not annotated, not reported with a caveat — withheld, with a named refusal saying the control false-positive rate was exceeded, and no arms returned at all.

The reasoning is that recall is only meaningful conditional on the gate being quiet when there is nothing to find. A gate that fires on everything has perfect recall and is worthless, and if you report recall without the control you have published a number that a broken gate maximises.

Our bound on that control rate is zero, which is stricter than most people expect. It is defensible because the corpus going into the control arm is one we have already accepted. If it trips a gate, either the gate is wrong or the corpus was never clean, and both of those need answering before any sensitivity figure means anything.

There is an implementation detail here that carried more weight than it looks like it should. When the measurement is withheld, asking for the recall figure raises rather than returning a null. A null gets formatted. Somebody writes a report template that renders a missing value as a dash or a zero or an empty cell, and the withheld case silently becomes a published one. Raising makes it impossible to put the withheld number in a document by accident, which is exactly the accident this whole system exists to prevent.

Attributing the catch

There is a subtlety that determines whether the experiment measures anything. Our gates emit blockers that name a problem but not the specific rule that caused it. So on a corpus with several injected defects, a fired blocker tells you the gate noticed something, and nothing about which thing.

The way out is isolation replay: to score whether defect number four was detected, replay the gate against the corpus with exactly that one defect applied. It is more expensive and it is the only version that produces an attributable answer.

We also record what we call off-target blockers — complaints the gate raised that were not the expected one. Those are not counted as detections. They are, however, kept, because a gate that reliably notices your injected defect for the wrong reason is a fact about your gate you want to know before you rely on it.

Densities, rounding, and the number that must not be zero

The curve is measured at several corruption densities, with a number of independently seeded instances at each. Two details from that machinery are worth stealing.

The first is that the requested density and the realised density are recorded separately. You ask for a quarter of the rules corrupted; a finite corpus gives you the nearest achievable fraction. Report the number you asked for and you have published a small lie in every row of the table.

The second is about rounding, and it is my favourite bug that never shipped. Converting a density into a count of items to corrupt requires rounding. The default rounding mode in many languages is banker's rounding — halves go to the nearest even number. At a low density on a small corpus, that can round the count down to zero. You then corrupt nothing, run the gate, find nothing, and publish a recall of exactly 0.0 for a gate that was never given anything to detect.

That is the defect class this entire series is about, appearing inside the machinery built to measure it. We use explicit half-up rounding, and where the count still lands at zero the arm reports a named refusal — the density is unrealisable on this corpus — rather than a recall figure.

Read a low number correctly

Suppose you run this and a gate comes back at 0.3. There are three quite different things that could mean, and they have different owners:

• The gate is narrow by design and the corruptor is broader than its remit. Fix the corruptor or accept the scope.
• The gate is blind in the region you sampled. Fix the gate.
• The defect is genuinely hard to detect statically and needs a different channel entirely. Stop investing in this gate and go build the other one.

Below a floor — ours is a half — we mark the arm as not sensitive, and the wording of that finding is deliberate: it says the gate is blind here. It very specifically does not say the corpus is clean. Those are opposite claims and a reader in a hurry will substitute one for the other unless the sentence forbids it.

Seeding is not a detail

Every arm's randomness is derived from a hash of the run seed, the instance identifier, and the arm identifier — never from position in a loop.

The reason is boring and the consequence is not: if the seed depends on iteration order, then adding a density to your sweep re-randomises every arm you already ran, and your curve changes without any change in the thing being measured. A detection curve you cannot reproduce is a detection curve you cannot defend, and you will be asked to defend exactly the one that came back low.

What to do with your own gates

Take your three most load-bearing checks. For each, write the smallest possible generator of the defect it claims to catch, and the name of the complaint you expect it to emit. Run the clean control first. Then inject and count.

You will learn one of two things, and both are worth the afternoon. Either you now have a number to put beside a check that previously had only a reputation — or one of the three catches nothing, has caught nothing for a long time, and everybody has been reading its silence as good news.

The one-line version

A gate that has never been given something to find has no measured detection rate, and a detection rate reported without a clean control arm is a number that a permanently-firing gate would maximise.