EU Law Search Benchmark: 102 Questions Tested
Summary
Pixel shares a benchmark of an EU law index against Exa web search using 102 new compliance questions spanning 41 EU acts. The index found the exact cited article first 76 times, compared with Exa’s 34, highlighting why article-level search matters for reliable legal research.
DEV Community
102 EU compliance questions we had never seen: our law index found the article first 76 times, a web search API 34
Yesterday we published a small test: 20 EU compliance questions, our EU law search against Exa's web search. Fair criticism of a 20-question test is that we might have tuned our ranking on exactly those 20. We did. So today we wrote 102 new questions and ran them before touching the ranking. The test 102 questions across 41 EU acts: GDPR, the AI Act, DSA, DMA, NIS2, DORA, MiCA, the VAT Directive, PSD2, the anti-money-laundering rules, consumer, product-safety and sustainability law. Each question names the article a correct answer must cite ("when must a company appoint a data protection officer" → GDPR Article 37). Every expected article was checked against the law's own text before either search ran. Ours counts only when the result is that article. Exa (search type auto, 5 results) counts when the title, URL or first 1,500 characters name the article number and the act, a deliberately lenient rule. Results Right article first Right article in top 5 Our EU law index 76 / 102 96 / 102 Exa search (auto) 34 / 102 75 / 102 On the 20 questions we did tune on, the picture is the same: ours 10 and 19, Exa 4 and 13. Why the gap A web search returns pages about the law: regulators' guidance, law-firm explainers, glossaries. Those are useful, but an agent that has to cite a provision needs the provision. Our index holds 47 EU acts article by article, in their latest consolidated text from the EU's own publications office, hosted in the EU. Where we lost Six questions had no right article in our top 5. Three were VAT questions: the VAT Directive's articles carry no headings in the official text, which our ranking leans on. "Health data" pulled up the European Health Data Space instead of GDPR Article 9. We will fix these against a fresh set of questions, not these 102. Limits, plainly We wrote the questions. They are checked against the law, not against either search, and they are public. The questions do not say "EU"; some of Exa's answers were UK or US pages, which a question naming the EU might avoid. English only so far. Try it The 122 questions with their expected articles and both scoring scripts (ours, Exa's) are in our public repository, so anyone can re-run them. If you already use Exa's SDK, point it at our base URL: the same search, contents and answer calls work, including structured answers, and EU-law questions are answered from the index. from exa_py import Exa
exa = Exa(api_key="YOUR_KEY", base_url="https://tools.funwords.eu/v1")
exa.search("when must a company appoint a data protection officer", category="law", num_results=5)
Every result carries the act, the article, its text and the EUR-Lex link. A free key gives 100 calls a month, no card; the full method and the company and research-paper results are on our benchmarks page, including the one where Exa is ahead. Send us your hardest compliance questions and we will add them to the next run.
Powered by Readybot.io
