The AI Crawler Access Checker shows which AI bots you block.

The AI Crawler Access Checker tests two separate things for each documented AI crawler: what your robots.txt asks it to do, parsed under RFC 9309 the way the crawlers themselves parse it, and what your server actually returns when a request arrives carrying that crawler’s user agent. The first is policy, the second is evidence, and the tool never merges them, because on many sites they disagree.

Because robots.txt is a request, not a lock. Since July 2025 Cloudflare has blocked AI crawlers by default on new domains, so a site can allow GPTBot in robots.txt while its CDN turns the same bot away at the edge with a 403, a challenge or a 402 payment demand. That gap, robots allows but the CDN blocks, is the single most common surprise this checker finds, and only a live request can reveal it.

No. GPTBot is OpenAI’s training crawler; ChatGPT search results come from a different bot, OAI-SearchBot, and pages a ChatGPT user asks about are fetched by a third, ChatGPT-User. Blocking GPTBot stops your content feeding future model training while leaving you visible in ChatGPT search. Blocking OAI-SearchBot is what removes you from ChatGPT search results. The checker reports each of the three separately.

No, and this is the most common misreading in AI crawler control. Google-Extended is a robots.txt token, not a crawler: it never appears in your logs, and it only governs whether your content trains Gemini models and grounds Gemini apps. AI Overviews are built from ordinary Google Search crawling by Googlebot, and there is no separate AI Overviews bot. The only controls that limit AI Overviews are snippet controls such as nosnippet and max-snippet, and noindex.

A user-triggered fetcher retrieves a page because a human asked an assistant about it, and most vendors treat that as the user browsing rather than a crawl. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores" them, and Meta and Amazon say much the same for theirs. The honourable exceptions are Anthropic’s Claude-User and Mistral’s MistralAI-User, which respect robots.txt even for user requests. The checker marks these rows "Robots can't block" so a robots.txt rule is never mistaken for a control; only a CDN or firewall can actually turn such a bot away.

The probe is evidence, not proof, and the checker says so on every result. It sends each bot’s documented user agent from RankX AI’s own servers, so an edge rule keyed on user agent strings shows up exactly. But a growing number of networks verify crawlers by IP address or Web Bot Auth, and those will refuse our probe while admitting the real crawler, which can read as blocked when the bot is fine. That is why every result also carries two control fetches, a normal browser and a plain curl, and why the verdict is always "as seen from our probe", never a guarantee.

Generic bot protection means your site accepted the browser control but refused the plain curl control and every bot user agent alike. When everything non-browser is blocked, no per-bot verdict is honest: the checker cannot tell whether GPTBot specifically is blocked or whether all automation is. The fix is to check your CDN’s bot settings, decide per purpose which AI crawlers you want, and allow the search-purpose ones explicitly if AI visibility matters to you.

That is a policy choice, not a technical one, and the checker deliberately does not make it for you. Blocking training bots such as GPTBot, ClaudeBot, meta-externalagent and CCBot keeps your content out of future model training without touching your visibility in AI search, because search runs on different bots. Around a quarter of the top 1,000 sites block GPTBot today. What the checker insists on is doing it per purpose: blanket-blocking everything with AI in the name also removes you from ChatGPT search, Perplexity and Claude’s citations, which is usually not what a marketing site wants.

ByteDance’s Bytespider and xAI’s Grok. ByteDance publishes no documentation for Bytespider at all, and third-party logs repeatedly report it ignoring robots.txt. xAI is worse: it documents no crawler token whatsoever, so robots.txt cannot even target it, and the names circulating online are unofficial and contradict each other. The checker shows both as "compliance unverifiable" rather than inventing a verdict, because a made-up green tick would be worth less than an honest unknown.

Add a robots.txt group per training bot and leave the search bots alone. Disallow GPTBot, ClaudeBot, meta-externalagent, Amazonbot, CCBot, MistralAI-Training, Google-Extended and Applebot-Extended, and do not add rules for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or DuckAssistBot. The page below the checker carries the exact snippet to copy. Then re-run the check, because a CDN rule can still override what robots.txt asks.

Only for some. Anthropic explicitly supports the non-standard Crawl-delay extension for its three bots. Amazon states Amazonbot does not support it, and Apple states Applebot ignores it too. Crawl-delay was never part of the robots.txt standard, so treat it as a polite request that a minority of crawlers honour, and use rate limiting at your server or CDN when you actually need to slow a bot down.

Anthropic publishes a single flat IP list covering ClaudeBot, Claude-SearchBot and Claude-User together, so log analysis can verify that a request really came from Anthropic but not which of the three bots sent it. OpenAI, by contrast, publishes a separate IP file per bot. When you are auditing your own logs, that difference decides how precise your attribution can be, and the checker’s roster table links each vendor’s verification file so you can check signatures yourself.

The target site sees a normal, small burst of GET requests for its homepage and robots.txt from RankX AI’s probe, identified per bot user agent, and nothing else: the checker never crawls beyond those two URLs and never republishes fetched content. Results are cached for 30 minutes per domain so repeat checks cost the target nothing, and the cache stores verdicts only. No account is needed and no personal data is kept.

Re-check after any CDN, WAF or robots.txt change, and quarterly otherwise, because the ground moves on both sides: vendors add bots and rename tokens through the year, and CDNs change defaults. Cloudflare default-blocked AI crawlers on new zones from July 2025 and has announced a further default change for ad-carrying pages from September 2026. A site that was open in spring can be silently closed by autumn without anyone touching robots.txt.