An empty checklist passes every audit
The most dangerous bug in an evaluation system is the one where silence and success are the same value. We found nine of them in one codebase, each behind a green test suite.
In almost every programming language, this is true:
all([]) == True"Every item in this empty list satisfies the condition." Logically impeccable. Vacuously so — there are no items, so none of them fail.
Now put that expression in a grader.
task_correct = all(check.passed for check in checks_that_apply)If checks_that_apply is ever empty, the task passes. Not "passes with a warning." Not "passes with zero coverage." Passes, identically to a task that satisfied every check you wrote.
This is the single most dangerous defect class in evaluation infrastructure, and it is nearly invisible to testing, because every test you write asserts what the code was designed to do and the defect lives entirely in what it was not designed to do.
Nine instances, one codebase
We went looking for this class deliberately, over several review rounds. We found nine distinct instances. Every one of them had shipped past a suite of roughly four thousand passing tests.
They all had the same shape, and it's worth seeing the range:
- A rubric criterion pointing at a grading gate that the specification never declared. The reference silently resolved to nothing, so nothing could fail.
- A grading gate that was declared but wasn't in the scored set. The answer key was written, stored, never compared.
- The identical bug on the state-checking side, discovered later, because the first fix had been written to say "answer" where it should have said "answer or state."
- An answer key whose type the checker couldn't read — a number handed to a set-comparison checker. The comparison never ran.
- A grading configuration that declared only progress milestones, which by design never enter the verdict. Every conjunct collapsed to a term that a do-nothing agent satisfies by construction.
- Two graders — one used at build time, one used on live runs — that enumerated the checked items differently, so a claim checked in one was skipped in the other.
The worst of them promoted cleanly through every quality probe we had. It shipped a certificate whose own recorded evidence read passed: false on one line and state_correct: true two lines below it.
Two of the nine were created by the fix to a previous one. That is the detail that matters most.
Why fixing them one at a time doesn't work
Each fix was correct, small, well-tested, and left the class alive somewhere else. The suite grew by ninety tests across those rounds and never caught a single new instance, because every added test asserted the corrected behaviour rather than the class of behaviour.
Once you've fixed the same bug three times in three places, you're no longer fixing bugs. You're playing whack-a-mole with a category, and the category is winning because it can appear anywhere a collection might be empty.
The way out is to stop writing fixes and write one statement of the invariant, enforced everywhere, that no local fix can satisfy on its own.
Where to look in your own system
The pattern to hunt is: a collection that can be empty, reduced to a verdict.
Concretely, go and read every place your code does one of these:
| Expression | The empty case |
|---|---|
all(...) over a filtered list | passes |
not any(...) over violations | clean |
| "no blockers found" | approved |
a lookup that returns None and is then skipped | never checked |
a loop with an else clause that runs when nothing broke | ran on zero items |
| percentage over a denominator that can be zero | 100%, or a crash |
For each one, ask a single question: can the collection be empty, and if it is, is the result a pass?
If yes, you have found one. The fix is almost never to flip all([]) to False — that would fail every legitimately empty case, of which there are usually many. The fix is to make emptiness a distinct outcome: a refusal, an error, an explicit "not measured," anything at all that is not the same value as success.
The design rule underneath it
Silence and success must never be the same value.
That principle generalises well beyond all([]). A configuration key that's absent should not mean the same thing as a key set to its permissive value. A missing measurement should not render as zero. An unchecked box should not look like a ticked one.
When a system cannot honestly produce a number, the correct output is not a number. It's a refusal that names what was missing and what would fix it. A refusal is annoying at build time and invaluable on a certificate; a wrong number is convenient at build time and indefensible everywhere else.
The organisational version
The same defect class runs through evaluation processes, not just evaluation code.
- A review checklist where every item is "N/A" gets signed off.
- A test plan whose coverage section is empty reads as complete.
- A vendor questionnaire where unanswered questions are counted as no concerns raised.
- A model card with no known limitations section, which reads as "no known limitations."
In each case, absence is being rendered as clearance. Any time your process converts "we didn't look" into "we looked and it was fine," you have the bug — and unlike the code version, nobody will ever get a stack trace.
What to actually do
Grep for it. Half an hour with your codebase and the table above will find candidates. It's the cheapest audit available.
Write the "does nothing" test. For any system that produces a verdict, construct the input that does nothing at all — the empty submission, the agent that takes no actions, the config with no rules — and assert it fails. That single test catches most of the class and almost nobody writes it.
Break your own checks on purpose. Stub out a check so it always passes, and confirm the tests that are supposed to protect it fail by name. If they don't, that check has no test — it has a test-shaped object beside it. We found several. Restore the check afterwards, and clear your build caches before you re-run, or you'll get a false result from stale bytecode and conclude the wrong thing.
Ask about it in vendor diligence. "Show me what your system does when a task declares a check that doesn't exist." The answer should be a loud refusal. If the answer is a pass rate, you now know something important about every number they've ever shown you.
The one-line version
The most expensive number in evaluation is a pass rate over an empty set of checks — because it looks exactly like a pass rate over a full one, and nothing in a green test suite will ever tell you which one you're holding.