Evaluation
When every retrieval method scores 1.000, you have measured nothing
On building evaluation sets that are capable of losing, and what a perfect tie actually tells you.
The result
I built a question-answering system over 7,462 NTSB aviation accident final reports. Quantitative questions route to guarded read-only SQL; narrative questions route to retrieval over report text with citations back to source records.
To decide how to do the retrieval half, I hand-labeled 17 questions with known-correct source documents and compared three approaches: BM25 lexical search, dense embedding search, and a hybrid of the two.
All three scored Hit@5 = 1.000.
My first instinct was that this was a good result. Every method finds the right document in the top five, so retrieval is solved and I can pick whichever is cheapest. I wrote it up as evidence that hybrid retrieval is "insurance for some query types rather than a universal upgrade."
That conclusion is not supported by that number. Nothing is.
Why a tie is not a finding
A comparison has discriminative power when it is capable of producing different answers for different methods. When three methods tie at a metric's ceiling, one of two things is true: the methods are genuinely indistinguishable, or the test cannot tell them apart. A perfect score cannot distinguish between those cases, because a saturated metric has no room left to express a difference.
Consider what Hit@5 = 1.000 on 17 questions requires. It requires that for every question, the correct document appears somewhere in the top five of a corpus of 7,462. If my questions contained distinctive terms that appear in the target report and almost nowhere else, then BM25 finds it on lexical overlap alone, dense retrieval finds it on semantic similarity, and the hybrid finds it because both of its components did. The methods did not agree because they are equivalent. They agreed because the questions were easy.
And I wrote the questions. Knowing the corpus. Which is exactly how you produce easy questions without intending to.
Seventeen is also too few
Set aside saturation for a moment. Even with a metric that had room to move, 17 items is not enough to separate retrieval methods whose real-world differences tend to be a handful of percentage points. A difference of two questions out of seventeen is 11.8 percentage points, and the confidence interval around a proportion at that sample size is wide enough to swallow most differences worth acting on.
Which means a small evaluation set fails in both directions. It cannot detect a real improvement, and it cannot rule one out. It produces numbers that feel like evidence and function like noise. Publishing "17 labeled evaluations" as a headline metric, which I did, advertises the weakness rather than the rigor.
Building a test that can lose
Make the metric harder. Hit@5 over a 7,462-document corpus is generous. Move to Hit@1, or to a rank-sensitive metric like MRR or NDCG@10 where being in position four instead of position one actually costs something. Reserve Hit@5 for a floor check, not for method selection.
Write questions designed to break specific methods. This is the part I skipped. Lexical search should struggle with paraphrase, so include questions that describe a failure mode without using the report's vocabulary. Dense retrieval should struggle with rare identifiers, so include tail registration numbers, part numbers, and exact dates. Both should struggle with multi-hop questions requiring two documents. A set built this way produces disagreement, and disagreement is the signal.
Get more items, cheaply. Hand-labeling is slow, which is why my set was small. But the reports themselves are structured: probable-cause statements can seed questions whose answer document is known by construction. That trades some realism for scale, and 200 semi-synthetic questions plus 30 careful hand-written adversarial ones is a far better instrument than 17 of anything.
Report uncertainty. Any proportion from a labeled set should carry a bootstrap interval. If the interval spans the other method's point estimate, say so, and say the comparison is unresolved.
What I should have concluded
The honest write-up is short: on a 17-question labeled set, BM25, dense, and hybrid retrieval all reached Hit@5 = 1.000. The set is too small and too easy to separate them, so this comparison does not support a choice between the three. The demo defaults to semantic retrieval and exposes all three methods for comparison because the evaluation does not support choosing hybrid as superior. That is an architectural decision under uncertainty, not a measured win.
That is less satisfying than a leaderboard. It is also the only claim the data supports, and it is the version that survives being asked about.
The broader habit is worth more than the specific fix. Before running any evaluation, I now ask what result would make me change my mind, and whether the instrument I have built is capable of producing it. An evaluation that cannot surprise you is not measuring your system. It is describing your assumptions back to you.
The project
Black Box AI is live over the NTSB corpus. The case study now states the saturation problem directly instead of drawing a conclusion the evaluation could not support.