RAG-on-Rails: Controlled-Experiment RAG Benchmark
Built a from-scratch RAG benchmarking pipeline with a controlled-experiment harness that isolates one variable at a time across 30 experiments (9 pipeline components) on a stratified dev set from MultiHop-RAG. Lifted complete-evidence retrieval (AllGold@10) from an 18.6% dense baseline to 49.2% using hybrid dense + BM25 retrieval with reciprocal rank fusion, a cross-encoder reranker, LLM query decomposition, and an agentic retrieve-judge-refine loop (Sonnet generates, Haiku judges). Scored every run with a deterministic, LLM-judge-free eval harness using verbatim golden-fact matching and custom metrics (AllGold@k, FactRecall@k, MRR, MAP), keeping results fast and fully reproducible.
Technical Deep-Dives
The Beginning
Why my two previous attempts at building a RAG pipeline stalled, what I learned about the importance of having a golden dataset to measure against, and how finding the right multi-hop benchmark, where answering a question requires connecting evidence across multiple articles, finally gave this project a foundation worth building on.
Setup
How I structured the RAG pipeline as a controlled experiment, changing one variable at a time, with YAML configs validated by Pydantic, a ChromaDB vector store that persists to disk across runs, and a file naming convention that makes every scorecard result self-explanatory without reading any code.
Metrics
Breaking down the six retrieval metrics that drive every experiment in this project: Hits@k, FactRecall, AllGold, MRR, and MAP, and why verbatim gold facts scraped directly from the source articles make fully reproducible, LLM-free evaluation possible.
The Baseline
Dense retrieval with BGE embeddings and a 450-token chunk size is the starting point every later experiment is measured against. The scorecard shows the retriever finds the first gold fact easily but consistently misses the bridging evidence that multi-hop questions depend on.
The Experiments
Thirty experiments across nine groups, each changing one variable at a time, to find what actually moves AllGold@10. The path from 18.6% to 39.8% was not the one I expected.
Agentic RAG
Every prior experiment used a static pipeline: one query in, a fixed number of chunks out. The agentic loop replaces that with iterative, evidence-grounded retrieval and pushes AllGold from 39.8% to 49.2%, at 14x the latency.
One Question, Three Systems
The analytical posts in this series measured aggregate metrics across hundreds of questions. This one does the opposite: it traces a single question through the baseline, the best static pipeline, and the agentic system, showing exactly which chunks each one retrieved, which gold facts each one found, and why the gap exists.