My retrieval benchmark passed the "replace all vectors with noise" test. So did a benchmark I broke on purpose.

There's a sanity check going around for retrieval evals: replace every vector in your index with random numbers, rerun the benchmark, see what it scores. If random garbage scores well, your benchmark was never measuring retrieval quality.

Good test. I ran it on my own production eval (1052 chunks, 218 files, 28 queries, baseline MRR@10 0.358). It passed.

Then I built a benchmark that is obviously broken — one where a random ranker scores 80% of what a perfect model scores — and ran the same test on that.

It passed too.

Correction 1: The floor is not zero

The version of this test I'd absorbed said a healthy noise floor should be "near zero." That's folklore and it will give you a wrong answer. Expected MRR under random ranking is fully determined by pool size N, gold count G, and cutoff k. Nothing else. For G=1 it's exactly H_k / N.

So "is 0.10 a bad noise floor?" is unanswerable in isolation. Compare measured against the analytic expectation, not against zero. The exact derivation for expected MRR under random ranking for G items in N candidates is:

E[MRR@k] = sum_{r=1}^{k} [ product_{j=0}^{r-2} (N-G-j)/(N-j) * G/(N-(r-1)) ] / r

Why one check isn't enough

Let's look at a clean toy corpus, 200 chunks, 20 queries, 20 random seeds:

Now shrink the candidate pool from 200 to 8. Change nothing else:

The leakage check passes on the broken benchmark, and it's right to pass — there is no leakage, the data is honest. The analytic expectation already accounts for pool size, so when the pool shrinks the expectation rises to meet the measurement.

But the benchmark is still worthless. The distance between "no system at all" and "perfect system" is just 0.19. Any real model lands in that sliver and run-to-run variance swamps the difference.

The Units Bug (How I failed my own test)

On my own eval, the first run reported 5.80x on Check 1. Apparent leakage. It turned out to be a bug in my test, not a finding: the simulation ranked 1052 chunks, but the scorer deduplicates to 218 file paths before computing MRR. The analytic assumed chunks, but the measurement was effectively over files.

I found it by inverting the approximation. E ≈ G·H₁₀/N, so N ≈ 1.39 × 2.929 / 0.02246 ≈ 181. Nowhere near 1052, close to 218.

Corrected: 0.82x on Check 1, 23.5x on Check 2, both pass.

One trap if your pipeline is hybrid

Run this with BM25 and any reranker disabled. Those components don't consume vectors. Leave them on and they'll carry the score, and you'll certify a benchmark you never tested.

What this establishes, and what it doesn't

Check 2 (23.5×) is a live measurement — real ONNX embeddings, real index, real queries. Check 1 (0.82×) is a simulation of the scoring harness: it assigns random scores to document IDs and confirms the scorer's arithmetic matches probability theory. It rules out counting bugs, dedup errors, and corrupted gold sets — it caught exactly such a bug on its first run. It does not push random vectors through the live index.

Two further limits: with 28 queries this detects gross benchmark failure, not mild leakage. And seven of those queries have more than one gold file, which raises their individual floors while carrying equal weight in a flat mean.

The Code

You can find the standalone, numpy-only rig here: github.com/gurukudte/eval-validity. It's completely self-contained, downloads no models, and runs in about a second.

It ships with the deliberately broken benchmark so you can watch Check 1 pass while Check 2 fails before pointing it at anything you care about. Swap the embed() function for your own pipeline; nothing else changes.