RAG Retrieval Showdown: Why Simple Reranking Won

ملخص

Pixel shares a measured case study of GigBid, a multi-agent freelance copilot built with LangGraph and three MCP servers. Across 180 simulated jobs, hybrid BM25 and dense retrieval with cross-encoder reranking outperformed requirement decomposition, offering practical lessons for RAG design and evidence-backed proposals.

DEV Community

The clever retrieval trick lost. A boring if statement won.

I built a multi-agent freelance copilot, measured it properly, and the results embarrassed my favourite idea. GigBid reads freelance job posts, decides whether to bid, and writes a proposal where every claim must link to evidence from the freelancer's own portfolio. A person approves everything before it is sent. I built it as a coursework project (LangGraph, three MCP servers, a React front end), and I measured it against a simulator I also wrote. This is not a tutorial. It is about the one result that changed how I think about where the effort should go. The idea I was proud of A job post asks for four things: PyTorch, Hugging Face, NLP, and some Python. Searching a portfolio with the whole post feels sloppy; surely it is better to decompose the post into requirements, retrieve evidence for each one separately, and merge the results. It is the standard advice in the advanced-RAG literature, it is satisfying to build, and I built it first. I wrote my hypothesis down before running anything (it is committed in docs/adr/0002-rag-hypotheses.md): decomposition would beat the naive baseline on jobs with four requirements by at least five points of Recall@5. What actually happened Evaluated on 60 stratified jobs per persona, three personas, evidence-level metrics against gold projects: variant Recall@5 MRR naive dense retrieval 0.91 0.63 hybrid BM25 + dense 1.00 0.74 hybrid + cross-encoder rerank 1.00 0.92 requirement decomposition, first version (merged per chunk) 0.84 0.42 requirement decomposition, fixed (merged per project) 0.92 0.45 Decomposition did not just fail to help. It made retrieval worse than the baseline, by a wide margin. My first implementation merged per-requirement rankings at the chunk level, which scatters a project's chunks across lists; merging at project level raised recall back up, but ranking quality stayed far below simple hybrid-plus-rerank. An ablation made the cause unmistakable. I took the full composed pipeline (decompose + BM25 + rerank + outcome boost), MRR 0.47, and removed one component at a time. Removing BM25, the reranker, or the outcome boost changed almost nothing. Removing decomposition alone restored MRR to 0.92. Why? Because on this kind of corpus every skill appears in several projects. A query for "Hugging Face" alone retrieves every project that mentions it; merging four such lists rewards projects that match one requirement strongly over projects that match all four. The whole-post query already contains the conjunction, and the cross-encoder is exactly the tool that reads it. Splitting the question threw that information away. I kept the failed variant in the results, and the chunk-level version's file as well. A negative result that is measured beats a win that is not. The thing that moved the needle more While I was tuning retrieval, a different number was quietly bad. My bid-or-skip decision had precision of about 0.85: it was bidding on jobs it should skip. I assumed the cause was something subtle, like weak evidence matching. It was not subtle. Twenty-four of the failures on the dev set were low-ball jobs: budgets of five to twenty dollars that my system happily bid on, because the decision rule never compared the budget to the freelancer's own rate floor. Fit was fine. Risk was fine. The job was just not worth the paid Connects. The fix was a dozen lines of deterministic code: if the posted budget cannot reach the freelancer's minimum, skip. Bid-or-skip F1 on that sample went from 0.89 to 0.98, and failures from 30 to 6. No model changed. No prompt changed. Set that next to the retrieval work: a day of retrieval variants improved a ranking metric (MRR 0.63 to 0.92), while the largest real error in the decision sat somewhere none of those variants touched, and a boring comparison fixed it. I did not measure how much retrieval quality moves the final decision, so I will not claim it does not. Retrieval quality is easy to measure, so it gets optimised; a missing business rule ...
RAG Retrieval Showdown: Why Simple Reranking Won
Powered by Readybot.io

الأحدث من Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …

Why a Java HPA Got Stuck at 20 Kubernetes Pods

Pixel shares how a Spring Boot pricing service autoscaled to 20 Kubernetes pods and stayed there because the JVM retained committed heap memory after traffic …