Web Search vs. API: Company Checks Benchmarked
Resumo
Pixel shares a benchmark comparing AI web search with a single API call for checking 25 companies across 10 countries. The announcement covers the initial results, four bugs found and fixed, the rerun, and cases where the API correctly returns unknown—useful insights for building reliable onboarding agents.
DEV Community
We lost our own benchmark the first time. Web search vs. one API call on 25 companies
If you build an agent that onboards suppliers or customers, sooner or later it has to check that a company is still registered, and that it isn't sanctioned or insolvent. We make an API that checks this against official registers, so we wanted to know if an agent needs it at all, or if a decent model with web search gets there by itself. So we ran a small benchmark, and the first run went badly for us: web search got 23 of 25 companies fully right, our API 17. This post covers how we measured it, the four bugs that run turned up, the rerun, and the cases where our API still answers "unknown". The setup We picked 25 companies from 10 countries. Thirteen are large, healthy and well known (Tesco, Novo Nordisk, Apple, Siemens and so on). The other twelve are the cases a check exists for: six are on a sanctions list, and the rest are dissolved, in liquidation or bankrupt (Carillion, Wirecard, Cambridge Analytica's UK company, Getin Noble Bank). For each company we wrote down three fields: register status (active, winding up or dissolved), sanctioned (on the UK, OFAC SDN or EU list) and insolvent. Each field was checked by hand against an official source, with the URL and the date. Where we couldn't verify a field, it isn't scored. That happened once, for the status of Lightyear's Dutch entity, because the Dutch register has no free access. Both arms use the same model (Claude Sonnet 5.5, low effort) and get the same question: The web arm has Anthropic's server-side web search and web fetch, at most 4 of each, with our own domain blocked. The API arm makes one POST to our counterparty-check endpoint with the registration number, LEI or name, and the model answers from the JSON it gets back. The system prompts differ in one way that matters. The API arm is told to answer only from the result and to say unknown where it's silent. The web arm is told to prefer official sources, and it's free to answer from what the model knows. Each answer ends with a small JSON object, and we compare the three fields with the ground truth by exact match. There's no model acting as judge. "Unknown" counts as not correct. Run 1: 23 to 17, and four bugs The web arm won the first run clearly. When we went through our misses, four of them were plain bugs on our side. UK insolvency only looked at The Gazette notices in our own copy, which starts in 2022. Cambridge Analytica (UK) Limited went into administration in 2018, so it came back with no insolvency at all. The check now reads the company's full insolvency history from Companies House as well. Garantex Europe, a sanctioned crypto exchange operator, has been deleted from the Estonian register. Our API answered its registry code with an HTTP 400, so the agent got nothing back, sanctions included. It now returns a report saying the company isn't in the register, and still screens it. Camaïeu in France and Wirecard in Germany came back as active, because Sirene and GLEIF still say active while a liquidation runs. An open liquidation notice that carries the company's number is now a fail. Notices that only match on the name, which is all the German register gives you, are flagged for review. Sanctions were matched on the name only. Garantex Europe has since been renamed, so its current name matches nothing, while the UK and OFAC entries both carry its Estonian registry code. A listing with the company's registration number or LEI now counts as a hit even when the name differs. Each fix got a regression test, and we ran the same 25 companies again. That makes run 2 an in-sample test for our API: we fixed it on these 25 companies, against a ground truth we wrote. A fair test needs companies we haven't looked at, and we haven't run one yet. Run 2: 19 to 20 Web search counterparty-check All three fields correct 19/25 20/25 Register status correct 20/24 24/24 Sanctioned correct 21/25 25/25 Insolvent correct 20/25 20/25 Sanctioned companies not flagged 2 of 6 0 of 6 Wrong answers / unknown 1 / 12 0 / 5 Cost p...

Powered by Readybot.io