DEV Community
Your page looks fine in Chrome. Here's what GPTBot actually receives
Open your product page in Chrome and it looks great: pricing table, feature list, customer quotes. Now ask a less flattering question: what does an AI crawler see when it asks for the same URL? For a lot of sites the honest answer is "much less than you think." Here's why, and a small script to check your own pages in a minute. Most AI crawlers don't run your JavaScript Googlebot renders pages: it fetches the HTML, then runs the JavaScript in a headless browser, then indexes the result. Most of the newer AI crawlers don't do that second step. Vercel looked at real crawler traffic across its network in late 2024 and found that the major AI crawlers, including OpenAI's and Anthropic's, fetched JavaScript files but didn't execute them. Whatever your page shows only after scripts run (a pricing table loaded from an API, reviews injected by a widget, tabs rendered client-side) simply isn't in what they read. Google's own AI features are the exception, because they lean on Google's rendering pipeline. Things may change, and crawlers don't publish their rendering behaviour in detail. But "the raw HTML is what they get" is the safe assumption for planning. That matters for AI answers because the engines quote what they can read. If your pricing only exists after hydration, an engine answering "how much does X cost?" has to get it from someone else's page. The check: fetch it like a bot, strip the scripts The script below asks for a URL with the published user agents of GPTBot (OpenAI's training crawler), OAI-SearchBot (the one behind ChatGPT search results) and PerplexityBot, throws away , , and blocks, and reports what's left: the HTTP status (some firewalls quietly return 403 to bot user agents), how many words of real text remain, the title, meta robots and canonical, whether the phrases you care about are actually in there. No dependencies, Python 3.8+. #!/usr/bin/env python3
"""What does an AI crawler actually receive from your page?
Usage:
python3 what_bots_see.py https://example.com/pricing "$29/mo" "free plan"
"""
import re
import sys
import urllib.error
import urllib.request
from html import unescape
User agents as published by OpenAI and Perplexity (checked 14 Nov 2026).
AGENTS = {
"GPTBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
"OAI-SearchBot": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot",
"PerplexityBot": "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",
}
def fetch(url, agent):
req = urllib.request.Request(url, headers={"User-Agent": agent})
try:
with urllib.request.urlopen(req, timeout=20) as r:
return r.status, r.read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
return e.code, e.read().decode("utf-8", "replace")
def visible_text(html):
html = re.sub(r"(?is)<(script|style|noscript|template)\b.*?</\1>", " ", html)
text = unescape(re.sub(r"(?s)<[^>]+>", " ", html))
return re.sub(r"\s+", " ", text).strip()
def find(pattern, html):
m = re.search(pattern, html, re.I | re.S)
return unescape(m.group(1).strip()) if m else "-"
def main():
if len(sys.argv) < 2:
sys.exit(doc)
url, phrases = sys.argv[1], sys.argv[2:]
for name, agent in AGENTS.items():
status, html = fetch(url, agent)
text = visible_text(html)
print(f"\n== {name}: HTTP {status}, {len(html):,} bytes of HTML, {len(text.split()):,} words of text")
print("title: ", find(r"]>(.?)", html))
print("meta robots:", find(r"]+name=[\"']robots[\"'][^>]+content=\"'", html))
print("canonical: ", find(r"]+rel=[\"']canonical[\"'][^>]+href=\"'", html))
for p in phr...
Powered by Readybot.io
External Link
Platform
You are about to visit:
https://example.comThis will open in a new tab. Make sure you trust this link.