LLM SEO: Make Your Site Visible to AI Assistants

DEV Community

LLM SEO: How to Make Your Site Visible to AI Assistants

Cross-posted from my own company blog: https://vouchgeo.com/blog/llm-seo. I'm the founder of Vouch, which is mentioned at the end. "LLM SEO" is one of several names for the same shift, next to AI SEO, GEO and AEO. Strip the label away and one practical question is left: when someone asks an assistant built on a large language model about your category, can it reach your pages, read them, and find something on them worth repeating? This guide stays on that practical layer: which crawlers to let in, what stops a model reading a page it was allowed to fetch, how to write pages it can quote, and how to tell whether any of it worked. If you want the bigger picture of why search is changing, AI SEO covers it, and GEO vs SEO vs AEO sorts out the terms. Reach, read, repeat It helps to treat LLM SEO as three tests a page has to pass, in order: Reach. The assistant's crawler or fetcher is allowed to request the page, and your server lets it through. Read. The response contains the content as text, not a blank shell waiting for scripts or a picture of a price list. Repeat. The text answers a buyer's question clearly enough that an assistant can quote or paraphrase it, and other sites back it up. A page that fails the first test never gets to the third, which is why the work starts with crawlers. Every provider fact below comes from that provider's own documentation, checked in November 2026; the rest is our editorial advice, to be tested against your own results. Reach: know which crawler does which job The large providers split their bots by job: training, search indexing, and fetches made because a user asked something. That split lets you decline training without disappearing from search answers. PROVIDER SEARCH TRAINING USER-TRIGGERED OpenAI OAI-SearchBot GPTBot ChatGPT-User Anthropic Claude-SearchBot ClaudeBot Claude-User Perplexity PerplexityBot none listed Perplexity-User A few details change how you act on that table: OpenAI lists four agents in its crawler documentation. The fourth, OAI-AdsBot, is limited to safety checks on landing pages that advertisers submit to ChatGPT. Blocking OAI-SearchBot keeps a site out of ChatGPT's search answers, apart from plain navigational links. Because ChatGPT-User acts on a person's request, OpenAI warns that robots.txt may not govern it, and where GPTBot and OAI-SearchBot are both allowed, OpenAI may reuse a single crawl for search and training. Anthropic says on its crawler page that its bots honour robots.txt, and that Claude-User lets site owners control which sites user-initiated requests can reach. Turning Claude-User or Claude-SearchBot off may reduce your visibility in Claude's answers. Perplexity says in its crawler guide that PerplexityBot does not gather content for training foundation models, and that Perplexity-User, acting for a user, generally ignores robots.txt. Our Perplexity SEO guide covers the firewall set-up Perplexity recommends. Google has no separate AI search crawler. AI Overviews and AI Mode use the normal Search index, and Google's AI features guide says a page needs to be indexed and eligible for a snippet to appear as a supporting link, with no extra technical requirements. Whether to allow training is a policy choice, and should you block GPTBot? walks through the trade-offs for each vendor. For visibility alone, the rule of thumb is short: let in the search crawler of every assistant your customers rely on. Each of the three vendors in the table publishes IP ranges for its agents, so firewall rules can admit the real crawlers and refuse impostors. To see where your robots.txt stands right now, run the free AI crawler checker. Read: make sure the fetch returns words A crawler that is allowed in can still come away with nothing useful. Common causes: Content that only appears after JavaScript runs. If the HTML your server sends is an empty frame, a crawler that does not run scripts sees an empty page. Serve the main text in the initial HTML. Facts locked in imag...

Powered by Readybot.io