Free X-Robots-Tag Header Auditor

Validate X-Robots-Tag HTTP headers across multiple URLs. Compare header directives against meta robots tags, detect conflicts, and see exactly which indexing signals your server is sending to search engines.

Why X-Robots-Tag Auditing Matters

The X-Robots-Tag HTTP header is one of the most overlooked sources of indexing directives. Unlike meta robots tags that live in HTML and are easy to spot, X-Robots-Tag headers are set at the server level and are invisible when viewing page source. A misconfigured server rule can silently add noindex to thousands of pages without anyone noticing.

Worse, when both X-Robots-Tag headers and meta robots tags exist on the same page, search engines combine the directives and apply the most restrictive interpretation. A page that appears indexable in its HTML source may actually be blocked by a header directive your team does not know about.

How to Use This Tool

1. Enter your URLs — paste up to 20 URLs, one per line. The tool accepts full URLs with or without the https:// prefix.
2. Click "Audit X-Robots Headers" — the tool fetches both the HTTP response headers and HTML content for each URL in parallel.
3. Review the comparison table — for each URL, you will see which directives are set in the X-Robots-Tag header versus the meta robots tag, with conflicts highlighted.
4. Check for conflicts — any mismatches between header and meta directives are flagged with severity levels and explanations.

How to Set the X-Robots-Tag Header

The X-Robots-Tag is configured at the web server level. Here are examples for common server environments:

Header set X-Robots-Tag "noindex, nofollow"

add_header X-Robots-Tag "noindex, nofollow";

X-Robots-Tag: googlebot: noindex
X-Robots-Tag: bingbot: noarchive

Common X-Robots-Tag Issues

• Server-wide noindex rules: A blanket X-Robots-Tag in the server config can accidentally block entire directories or file types from indexing.
• CDN or proxy injection: Cloudflare Workers, AWS CloudFront functions, or reverse proxies can inject X-Robots-Tag headers without developers being aware.
• Staging config in production: Server configurations from staging environments with noindex headers can persist when deploying to production.
• Conflicting header and meta signals: When both sources exist, search engines apply the union of all directives, making the most restrictive one win.
• Non-HTML resource blocking: PDFs, images, and other resources can only be blocked via X-Robots-Tag since they cannot contain meta tags.

X-Robots-Tag vs Meta Robots: Key Differences

Frequently Asked Questions

What is the X-Robots-Tag HTTP header?

The X-Robots-Tag is an HTTP response header that provides search engine crawling and indexing directives. It supports the same directives as the HTML meta robots tag — including noindex, nofollow, noarchive, and nosnippet — but is set at the server level rather than in page markup. This makes it essential for controlling indexing of non-HTML resources like PDFs, images, and downloadable files that cannot contain meta tags.

What happens if X-Robots-Tag and meta robots conflict?

Search engines combine directives from all sources and apply the most restrictive interpretation. If the X-Robots-Tag header says "noindex" but the meta robots tag says "index, follow", the page will NOT be indexed because the noindex directive from the header takes precedence. This combination behavior is why auditing both sources together is critical.

How do I set an X-Robots-Tag header on my site?

The configuration depends on your server. In Apache, add Header set X-Robots-Tag "noindex" to your .htaccess file. In Nginx, use add_header X-Robots-Tag "noindex";. In Next.js, you can set headers via the headers() config in next.config.js or via middleware. Cloudflare users can set it through Transform Rules.

When should I use X-Robots-Tag instead of meta robots?

Use X-Robots-Tag when you need to control indexing for non-HTML resources like PDFs, images, or video files that cannot contain meta tags. It is also useful when you want to apply directives to many pages at once via server configuration without modifying page templates, such as blocking an entire staging subdomain or a specific directory.

Does Google respect the X-Robots-Tag header?

Yes. Google fully supports the X-Robots-Tag HTTP header and treats it the same as meta robots directives. Google processes the header for all resource types including HTML pages, PDFs, images, and other files. Bing also supports the X-Robots-Tag header.