Four AIs Tested on the Same Hidden Coding Suite

DEV Community

I Gave Four AIs the Same Spec. All Four Aced It.

I ran the fourth file through the test suite and watched 39 dots print out. Again. ....................................... [100%]
39 passed in 0.05s
Same output as the third file. Same as the second. Same as the first. Four different AIs, one spec, one hidden suite — and not one of them dropped a single test. I had built the suite to bite. It bit the three broken versions I wrote to check it, exactly where they were broken. It did not bite a single AI. The claim on the table Every week brings a new leaderboard saying which AI writes the best code. The numbers come from benchmarks with names and versions, run by the labs or by people with strong opinions and a scoring script. I have never trusted a leaderboard I cannot re-run, so this series does the small, checkable version instead: one concrete job, one spec written in plain words, one test suite nobody showed the models, first drafts only. The tests decide. Everything else is colour. The question for this episode: you hand four AI tools the same spec and nothing else — which one's first draft actually passes? The lab The job is a small module called flatbill. It splits shared-flat bills: an even split, a weighted split, and a late fee. All money is integer paisa, standard library only, three functions, no floats anywhere. I picked this job because it is real work — the kind of thing a small app actually needs — and because it is full of the small decisions where code quietly goes wrong. Leftover paisa have to land somewhere exact: one each, to the first names alphabetically. The weighted split has a tie-break rule: largest fractional remainder wins, ties broken by name. A weight of 0 pays exactly 0 and never gets a leftover paisa. The late fee has a 3-day grace period, then 1% per day, rounded half-up (10.5 becomes 11, not 10), capped at 20% rounded down. And there is a leap-year test in there, because someone always forgets February. None of these are hard. All of them are easy to get slightly wrong. That is the point. The suite is 39 tests, written before any AI saw the spec, frozen at 21:41 on 12 November 2026 with its hash recorded (82a72d29…e553a47 — the full hash is in the data pack). The AIs never saw the tests. They got the spec and nothing else, the same words pasted verbatim, one fresh chat each, default model, one shot each. I did not paste failures back. I did not ask for fixes. None of them asked me a question, so the standard nudge I had ready — "use your best judgement and deliver the file" — never got used. Before the runs, I checked the suite could actually fail things. I wrote three broken versions of the module on purpose: one that rounds naively and never fixes the remainders, one with no grace period, one that breaks ties by input order, rounds half-to-even and skips the cap. They scored 32, 33 and 34 out of 39 — each failing only its own bug's tests. The suite discriminates. It just never got the chance to discriminate between the AIs. One more honest detail from before the freeze: the suite caught a mistake of mine. A remainder test used 1001 paisa over 3 people and expected one leftover paisa. Wrong — 1001 % 3 is 2, and the expected dict summed to 1000, not 1001. My reference implementation produced the correct answer and exposed it. I fixed the test before the freeze, and that was the last change the suite ever got. What happened The runs happened on Thursday 12 November, between 21:43 and 22:00, back to back in one sitting. The models themselves were fast. Copilot answered in eleven seconds. ChatGPT, 21:43. Under a minute, one code block, no explanation, no questions. One thing I cannot tell you about this run: the page showed no model name anywhere. The responses were labelled only "ChatGPT". So the episode cannot name its model, and I will not guess. Claude, 21:45. Under a minute, one code block, no explanation. Model shown: Sonnet 5.5 Medium. Copilot, 21:51. Eleven seconds, prompt to complete file. New conversation in the Copilot chat web U...

Powered by Readybot.io