Free Robots.txt Generator

Generate a production-ready robots.txt file in seconds. Configure default crawling rules, block specific AI crawlers like GPTBot and ClaudeBot, add your sitemap URL, and write custom directives — all with a live preview that updates instantly.

What Is a robots.txt File?

A robots.txt file is a plain-text file placed at the root of your website (e.g. https://example.com/robots.txt) that instructs web crawlers which pages or sections of your site they are allowed or not allowed to access. It follows the Robots Exclusion Standard and is one of the first files search engine bots check when they visit your domain.

The file uses simple directives: User-agent specifies which bot the rule applies to (use * for all bots), and Allow or Disallow controls access to paths.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Blocking AI Crawlers: Why It Matters

AI companies train their large language models on content scraped from the web. If you want to prevent your content from being used to train AI systems — or simply control how your intellectual property is used — you can add Disallow directives for specific AI bots:

• GPTBot — OpenAI's crawler used to train ChatGPT and other models
• ClaudeBot — Anthropic's crawler used to train Claude models
• PerplexityBot — Perplexity AI's web crawler for its AI search engine
• CCBot — Common Crawl bot whose datasets are widely used in AI training
• Google-Extended — Google's opt-out mechanism for Bard and Vertex AI training

Note that robots.txt is a voluntary standard — ethical crawlers respect it, but it does not technically prevent access. For stricter enforcement, combine robots.txt with authentication or IP blocking.

robots.txt Best Practices

• Always include a Sitemap directive — Helps search engines discover all your pages efficiently
• Test before deploying — Use Google Search Console's robots.txt tester to validate your file
• Don't block CSS or JS — Blocking stylesheets and scripts prevents Google from rendering pages correctly
• Disallow doesn't mean noindex — Blocked pages can still appear in search results if linked from other sites; use noindex meta tags for that
• Keep it simple — Only add rules that serve a purpose; unnecessary complexity causes bugs
• Use specific paths — Block /admin/ rather than broad patterns to avoid accidentally blocking important pages