Benchmarks with receipts
Retrieval quality here is measured, not asserted — the harness, the golden set, and the baselines are all committed to the repo, and CI runs the regression gate.
The result
Mean recall@20 on the golden set, full 597K-point corpus, measured live at each stage:
| Retrieval configuration | Mean recall@20 | Hit rate | Zero-result queries |
|---|---|---|---|
Dense-only (OpenAI text-embedding-3-small) |
0.44 | 0.74 | 13 / 50 |
| + BM25 sparse, server-side RRF fusion | 0.59 | 0.84 | 8 / 50 |
| + cross-encoder reranking (MiniLM-L-6) | 0.61 | 0.86 | 7 / 50 |
The headline: dense-only retrieval returned nothing relevant for 26% of realistic analyst queries — mostly ones naming exact tokens like development codes (ABX464, BBO-10203, REGN475/SAR164877) that embeddings blur. Hybrid fusion cut that nearly in half and lifted mean recall by a third.
Methodology (and why it isn't circular)
The golden set (eval/golden.jsonl, 50 queries) is generated by eval/build_golden.py:
- Ground truth comes from exact payload matching — server-side Qdrant filters on literal intervention/condition strings. The vector retriever being evaluated plays no role in defining relevance.
- Queries target small expectation sets (2–15 trials globally), so recall@20 is meaningful — a query with 900 relevant trials can't be "recalled" into 20 slots.
- Queries are natural-language templates over the ground-truth entities ("Which trials use X?", "trials evaluating X in patients with Y"), never the filter syntax itself.
- The eval calls
research_agent.retrieve_trials()— the exact function the production agent uses — so the eval and the product cannot drift apart.
Reproduce it
# against your own seeded corpus
uv run python eval/run_eval.py
# regression gate (skips cleanly if no corpus is reachable)
uv run pytest eval/
# regenerate the golden set from scratch (deterministic, seeded)
uv run python eval/build_golden.py
eval/baseline.json holds the committed baseline; improving retrieval means beating it and deliberately re-recording (--write-baseline), which makes every quality change a reviewable diff.
Known honest caveats
- 50 queries is a regression harness, not an academic benchmark; the point is that changes are measured on a fixed, committed set.
- Ground truth by exact lexical match slightly understates recall (case-variant duplicates of the same drug name aren't counted as expected).
- The reranker adds real latency (~1.2s per 100 candidates on an M-series CPU, more on small cloud vCPUs);
RERANKER_PROVIDER=nonedisables it if you'd rather have the speed.