Free AI Crawler Access Checker
Enter any URL to see which AI crawlers your robots.txt lets through. The tool checks 50 documented crawler tokens — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Copilot and the rest — against the exact path you enter, and shows the rule that decided each verdict.
Checks the exact path you enter. Give it a blog post URL to see whether that page is reachable, not just your homepage.
Why AI crawler access matters
Before a language model can mention your brand in an answer, something has to fetch your page. Sometimes that is a training crawl that happened months ago; more often, for the answers being generated today, it is a live fetch made while the user waits. A single line in robots.txt decides whether that fetch is allowed.
The mistake we see most often is a blanket block written for one reason — usually a decision about model training — that also removes the site from live answers. Those are different crawlers with different tokens, and they deserve different answers.
The four kinds of crawler
Blocking one of these is not the same decision as blocking another:
- AI answers — live fetches made to answer a user right now (ChatGPT-User, Claude-User, Perplexity-User). Block these and you disappear from answers being generated today.
- Indexing — search and answer-engine index crawls that feed retrieval (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot). Block these and you are not in the pool of pages an engine can pick from.
- Training — crawls collecting data for future model training (GPTBot, Google-Extended, Applebot-Extended, CCBot). Blocking these is a legitimate content policy decision and does not affect whether you are cited today.
- Other AI — ads, cloud tooling and general-purpose AI crawlers (OAI-AdsBot, Google-CloudVertexBot, Diffbot).
How to use this tool
- Enter a real content URL — not just your homepage. robots.txt rules are path-specific, so
/blog/can be blocked while/is open. - Read the answer and indexing rows first — those are the crawlers that cost you citations when blocked.
- Check the deciding rule — every verdict shows the exact
AlloworDisallowline and theUser-agentrecord it came from, so you know which line to edit. - Look at the noindex warning — a page can be crawlable and still be excluded from answers by a noindex directive.
Common issues
- A staging block that shipped to production: a
Disallow: /underUser-agent: *blocks every crawler in the list at once. - A CDN or security template you did not write: several managed hosts and WAF products ship an AI-crawler block by default. If you never edited robots.txt and the answer crawlers are blocked, this is usually why.
- Assuming one token covers a family: blocking Googlebot does not block Google-Extended, and blocking Applebot does not block Applebot-Extended. Each token needs its own record.
- Confusing crawl access with indexability: allowing a crawler in robots.txt does nothing if the page returns a noindex header.
- A robots.txt that returns a server error: worse than having no file at all — Google treats a 5xx as a full block for the first 12 hours.
Frequently Asked Questions
Which AI crawlers does this tool check?
It checks 50 documented AI and search crawler user agents, including OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot), Perplexity, Google (Googlebot, Google-Extended, Google-NotebookLM), Microsoft Copilot, Apple, Amazon, Meta AI, xAI Grok, Mistral, DeepSeek, Qwen, Kimi, Cohere, Common Crawl CCBot and more.
Should I block AI crawlers?
It depends on which one. Blocking training crawlers like GPTBot, Google-Extended or Applebot-Extended keeps your content out of future model training and costs you nothing in citations today. Blocking answer crawlers like ChatGPT-User, Claude-User or PerplexityBot is different — those fetch pages to answer a question right now, so blocking them removes you from answers users are asking for. This tool separates the two so you can make each decision on its own.
Why does blocking Applebot not block Applebot-Extended?
They are separate product tokens. Applebot crawls for Apple search features; Applebot-Extended is the opt-out token for training Apple foundation models. robots.txt matches user-agent tokens exactly, so a rule naming Applebot does not govern Applebot-Extended. The same applies to Googlebot and Google-Extended. This tool follows the exact-match rule from RFC 9309 rather than substring matching, which is why the two can show different verdicts.
My robots.txt allows everything but I am still not cited. Why?
Crawler access is necessary, not sufficient. Check three other things: whether the page carries a noindex directive (this tool flags that), whether your content is rendered server-side — most AI crawlers do not execute JavaScript — and whether the page actually answers the question in a form a model can quote. Our AEO Grader scores that last part.
What happens if my robots.txt is broken or times out?
This tool tells you rather than guessing. An unreachable robots.txt is not the same as a permissive one — and it is actively worse than a missing one, because Google treats a robots.txt returning a 5xx status as "block everything" for the first 12 hours before falling back to treating it as absent.
Enter your website to track its AI visibility across ChatGPT, Gemini, Claude, and Perplexity — and turn chatbot mentions into traffic.
- Set up in minutes
- Cancel anytime
Related Tools
See whether your site appears in Common Crawl, the open corpus behind most AI models.
AEO GraderScore a page on how easily an answer engine can quote and cite it.
Robots.txt ValidatorCheck your robots.txt for syntax errors and conflicting directives.
Robots.txt GeneratorBuild a robots.txt file with explicit rules for AI and search crawlers.