Robots.txt Rules May Block AI Search Crawlers

ملخص

Pixel’s announcement explores how robots.txt rules can unintentionally block AI search crawlers that could cite pages and send visitors. An analysis of 700 top-ranked sites found that 43.6% of sites blocking OAI-SearchBot did so through broad rules rather than an explicit choice, highlighting how RFC 9309 group behavior can affect visibility.

DEV Community

Your robots.txt probably does not say what you think it says about AI crawlers

I read 700 of the highest-ranked sites' robots.txt files on 9 November 2026, matched each one against the AI crawler tokens their operators document, and published the whole reading — domain by domain, with the exact line that decided each verdict — as open data. 407 of the 700 files could actually be read; 145 hosts refused the request outright. Those are large, high-ranked sites, not small businesses, so do not read the shares below as a picture of small business. Two numbers from that reading are the reason for this post. Of the sites that block GPTBot, 80.3% wrote a rule that names it. That block was a decision: somebody sat down and kept their pages out of a training corpus. Of the sites that block OAI-SearchBot — the crawler that fetches a page at the moment someone asks a question, in order to answer with it and link back — 43.6% never chose it at all. The block came from a rule aimed at every bot alike. That is the whole problem in one sentence: the file people wrote to keep out training crawlers is also keeping them out of answers that would have sent them visitors. And it usually happens through one of four readings of RFC 9309 that are easy to get backwards. 1. * is a fallback, not an addition This is the one that bites hardest, in both directions. User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /
The GPTBot group does not inherit Disallow: /admin/. If any group names the crawler, the * group is ignored for it entirely — not merged, not consulted for paths the specific group is silent about. So a file like this: User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /drafts/
allows GPTBot into /private/. Everything you meant to close for everyone, you have just opened for the one crawler you were paying attention to. The inverse is the same rule and it is why a generator that writes a group per crawler has to say so: adding User-agent: GPTBot to a file whose * group disallows half the site relaxes the policy for GPTBot unless you repeat those paths inside its own group. 2. Group selection is exact, not a prefix A group for Google-Extended does not govern Googlebot. A group for Googlebot does not govern Googlebot-Image. The matcher compares the product token itself, case-insensitively, and nothing else. Every parser I tried this against got it right. Every human reading I have seen of a file like User-agent: Google-Extended
Disallow: /
got it wrong in the same direction: people read it as "I have blocked Google". It blocks one Google product, the one used for Gemini training, and leaves the search index untouched. Which, for most sites, is exactly what they wanted — they just did not know they had got it. 3. Longest match wins, and Allow breaks the tie Disallow: /blog/
Allow: /blog/public/
/blog/public/x is allowed: the longer pattern wins. And when an Allow and a Disallow match with the same length, the Allow wins. People write the pair above expecting the first line to dominate because it came first; order does not enter into it. 4. An empty Disallow: allows everything User-agent: GPTBot
Disallow:
That is the documented way to say "no restrictions". Read as a zero-length prefix — which every path matches — it would mean the opposite, and a file that was meant to open everything would close everything. If you write a matcher yourself, this is the test case to add first. What I ended up building The answer a site owner actually wants is not "allowed" or "blocked". It is "allowed or blocked, because of this line, in this group" — because that is the only form of the answer they can check against their own file. Every parser I looked at returns the boolean and throws the provenance away, so our matcher returns the deciding rule, and the free checker quotes it with its line number. Two free things came out of it, no sign-up on either: The reference dataset of AI crawler user agents — the token as it appears in User-agent, who operates it, what it fetches pages for, and what its operator d...
Robots.txt Rules May Block AI Search Crawlers
Powered by Readybot.io

الأحدث من Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …

Why a Java HPA Got Stuck at 20 Kubernetes Pods

Pixel shares how a Spring Boot pricing service autoscaled to 20 Kubernetes pods and stayed there because the JVM retained committed heap memory after traffic …