RAG Retrieval Showdown: Why Simple Reranking Won
ملخص
Pixel shares a measured case study of GigBid, a multi-agent freelance copilot built with LangGraph and three MCP servers. Across 180 simulated jobs, hybrid BM25 and dense retrieval with cross-encoder reranking outperformed requirement decomposition, offering practical lessons for RAG design and evidence-backed proposals.
DEV Community
The clever retrieval trick lost. A boring if statement won.
I built a multi-agent freelance copilot, measured it properly, and the results embarrassed my favourite idea. GigBid reads freelance job posts, decides whether to bid, and writes a proposal where every claim must link to evidence from the freelancer's own portfolio. A person approves everything before it is sent. I built it as a coursework project (LangGraph, three MCP servers, a React front end), and I measured it against a simulator I also wrote. This is not a tutorial. It is about the one result that changed how I think about where the effort should go. The idea I was proud of A job post asks for four things: PyTorch, Hugging Face, NLP, and some Python. Searching a portfolio with the whole post feels sloppy; surely it is better to decompose the post into requirements, retrieve evidence for each one separately, and merge the results. It is the standard advice in the advanced-RAG literature, it is satisfying to build, and I built it first. I wrote my hypothesis down before running anything (it is committed in docs/adr/0002-rag-hypotheses.md): decomposition would beat the naive baseline on jobs with four requirements by at least five points of Recall@5. What actually happened Evaluated on 60 stratified jobs per persona, three personas, evidence-level metrics against gold projects: variant Recall@5 MRR naive dense retrieval 0.91 0.63 hybrid BM25 + dense 1.00 0.74 hybrid + cross-encoder rerank 1.00 0.92 requirement decomposition, first version (merged per chunk) 0.84 0.42 requirement decomposition, fixed (merged per project) 0.92 0.45 Decomposition did not just fail to help. It made retrieval worse than the baseline, by a wide margin. My first implementation merged per-requirement rankings at the chunk level, which scatters a project's chunks across lists; merging at project level raised recall back up, but ranking quality stayed far below simple hybrid-plus-rerank. An ablation made the cause unmistakable. I took the full composed pipeline (decompose + BM25 + rerank + outcome boost), MRR 0.47, and removed one component at a time. Removing BM25, the reranker, or the outcome boost changed almost nothing. Removing decomposition alone restored MRR to 0.92. Why? Because on this kind of corpus every skill appears in several projects. A query for "Hugging Face" alone retrieves every project that mentions it; merging four such lists rewards projects that match one requirement strongly over projects that match all four. The whole-post query already contains the conjunction, and the cross-encoder is exactly the tool that reads it. Splitting the question threw that information away. I kept the failed variant in the results, and the chunk-level version's file as well. A negative result that is measured beats a win that is not. The thing that moved the needle more While I was tuning retrieval, a different number was quietly bad. My bid-or-skip decision had precision of about 0.85: it was bidding on jobs it should skip. I assumed the cause was something subtle, like weak evidence matching. It was not subtle. Twenty-four of the failures on the dev set were low-ball jobs: budgets of five to twenty dollars that my system happily bid on, because the decision rule never compared the budget to the freelancer's own rate floor. Fit was fine. Risk was fine. The job was just not worth the paid Connects. The fix was a dozen lines of deterministic code: if the posted budget cannot reach the freelancer's minimum, skip. Bid-or-skip F1 on that sample went from 0.89 to 0.98, and failures from 30 to 6. No model changed. No prompt changed. Set that next to the retrieval work: a day of retrieval variants improved a ranking metric (MRR 0.63 to 0.92), while the largest real error in the decision sat somewhere none of those variants touched, and a boring comparison fixed it. I did not measure how much retrieval quality moves the final decision, so I will not claim it does not. Retrieval quality is easy to measure, so it gets optimised; a missing business rule ...

Powered by Readybot.io