SpecGuard: Can AI Agents Test Code Without Approving Bugs?

Краткое содержание

Pixel shares research on SpecGuardBench, which tested how well AI coding agents build validation harnesses from natural-language requirements. The best agent was faithful on just 63% of tasks, and buggy reference code reduced faithfulness by 19 to 31 points—highlighting why tests must check specifications, not simply copy existing behavior.

DEV Community

SpecGuard: When AI Coding Agents Write Tests That Approve Bugs

Ask a coding agent to write checks for a requirement and it will often copy what the existing code does instead. In SpecGuardBench, the best agent wrote a faithful validation harness for only 63% of tasks. When the reference code was buggy, the agents' faithfulness fell by 19 to 31 points, because they wrote checks that approved the bug. Here is a free, in-browser way to test any model for this. The problem: tests that agree with the code Here is a one-line requirement: slugify(title) MUST return a string that contains only lowercase letters a–z, digits 0–9 and single hyphens, with no leading or trailing hyphen. You give a coding agent this requirement, the current implementation and the existing test, and ask it for a validation harness: a set of checks that tells you whether an implementation meets the requirement. The easy way out is to run the current implementation, record its outputs and assert that any other implementation returns exactly the same thing. That harness passes on the current code and looks thorough. It is also wrong in two ways: It approves whatever the reference does. If the reference has a bug, the harness treats the bug as correct behaviour. It rejects valid code. The requirement says nothing about accented letters. One correct implementation turns "Café au lait" into caf-au-lait, and another turns it into cafe-au-lait. Both meet the requirement. A harness that copies the reference fails the second one. This matters more as agents write more of the code. The harness is what decides whether the agent's next change is accepted. If it checks the old code instead of the requirement, it blocks valid changes and lets real bugs through. What the research found SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications? (arXiv:2610.16870) tested this directly. The benchmark has 291 Python tasks covering eight kinds of verification logic. For each task there is a correct implementation, a buggy one that breaks the requirement but still passes the existing test, and valid alternatives that behave differently wherever the requirement says nothing. A harness counts as faithful only if it rejects the bug and accepts every correct version. Coding agent (paper, 291 tasks) Faithful Catches the bug Rejects correct code OpenCode / GLM-5.3 62.9% 74.2% 15.1% Codex / GPT-5.6-sol 61.9% 81.8% 23.0% Claude Code / Claude Opus 5 50.2% 65.3% 20.6% OpenCode / DeepSeek-V4-Pro 45.0% 66.7% 26.8% Then the authors set a trap. They showed the agents the buggy implementation as the reference. If an agent were really working from the requirement, the reference wouldn't matter. It mattered a lot: Agent (paper, 100-task Lite set) Correct reference shown Buggy reference shown Codex 65% 46% OpenCode / GLM 68% 39% OpenCode / DeepSeek 52% 28% Claude Code 52% 21% Faithfulness fell by 19 to 31 points. The agents trusted the code they were shown more than the requirement they were given. The solution: SpecGuard, a benchmark that runs in your browser SpecGuard is a small, free re-creation of SpecGuardBench that runs entirely in a browser tab. It ships as a tab in ZeeSpec, an open-source spec builder for AI coding agents, next to the EPIC Score tab that grades the spec itself. It has 16 JavaScript tasks, two for each of the paper's eight types of verification logic: Type What the check must establish Functional postcondition The return value satisfies a rule about the inputs. Input validation Certain inputs raise an error or give a specified result. State postcondition The state after the call relates to the state before it as required. Frame condition The call leaves specified state unchanged. Interaction protocol Other components are called with the right arguments, the right number of times. Call order Events within one run happen in the specified order. Interface signature The interface has the specified names, number of arguments or members. Relational property Two or more runs relate to e...
SpecGuard: Can AI Agents Test Code Without Approving Bugs?
Powered by Readybot.io

Последнее от Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …

Why a Java HPA Got Stuck at 20 Kubernetes Pods

Pixel shares how a Spring Boot pricing service autoscaled to 20 Kubernetes pods and stayed there because the JVM retained committed heap memory after traffic …