Crawlability Header Inspector — Check X-Robots-Tag and Meta Robots

An X-Robots-Tag checker inspects robot directives returned in HTTP response headers and compares them with robots meta tags in the page’s HTML. Enter a permitted public URL to request its response through the site’s restricted inspection endpoint, or paste HTML for a clearly labeled offline analysis. The report distinguishes the final response from redirect hops and shows index-related and link-following directives with their declared crawler scope. A noindex directive can restrict indexing, but its absence does not guarantee that a page will be indexed. The checker cannot confirm search-engine rendering, canonical selection, internal discovery, or a crawler’s exact decision. It can, however, identify conflicting markup and recorded response signals that need targeted inspection.

X-Robots-Tag Checker At a Glance

Where Robots Directives Are Declared

Indexing instructions can appear in an HTTP header or within HTML as a meta tag. The X-Robots-Tag header can apply to HTML responses and to resources such as PDFs, while a robots meta tag is part of a document head that a crawler must retrieve to interpret it. A page might contain meta name="robots" content="noindex,follow", for example, or return X-Robots-Tag: noindex at the HTTP layer. These signals are not interchangeable with robots.txt. A robots.txt Disallow rule restricts crawling access; a crawler that cannot fetch the page may never observe a noindex meta directive placed inside it. The inspector keeps these sources distinct so a technical review does not collapse several independent mechanisms into one vague crawlability score.

What an X-Robots-Tag Response Means

HTTP headers belong to the specific response that returned them. A request that redirects might produce a different set of headers at each hop, while a final HTML page can carry another set. The checker records the redirect history supplied by the endpoint, then analyzes the final page’s header values and HTML. A header may also be scoped to a named crawler rather than all crawlers. For this reason, the tool preserves the apparent agent label beside the directive instead of erasing its context. Always check whether a CDN, application server, or proxy injects headers conditionally based on request path, user agent, or deployment environment before concluding that the observed response is universal.

Noindex, Index, and None Directives

The noindex directive tells compliant crawlers not to include a fetched resource in search results, while index is an affirmative directive that may be redundant under default behavior. The shorthand none generally represents noindex together with nofollow. The checker flags cases where index and noindex or follow and nofollow appear together, because contradictory values deserve manual interpretation. It does not choose a winner on your behalf. The absence of noindex is only a missing restriction in that response, not evidence that the URL is indexable in practice. HTTP errors, inaccessible content, duplication, canonical signals, and search-engine selection can still prevent an apparently unrestricted page from appearing in results.

Worked Example: Header and Meta Conflict

Suppose the server returns X-Robots-Tag: noindex while the HTML contains meta name="robots" content="index,follow". An inspection that looks only at page source might incorrectly conclude that indexing is permitted. A header-only inspection could miss a second directive inside the document. In the combined report, both values remain visible and the conflict is flagged for investigation. The next step is to identify which layer generated each instruction, determine whether the header applies to the target crawler, and remove accidental disagreement. A single successful audit should be repeated after cache invalidation or deployment because stale edge responses can temporarily disagree with updated HTML.

How to Inspect a Public URL

Enter a complete HTTP or HTTPS URL and start the inspection. The browser sends the URL to the existing restricted PHP endpoint rather than fetching arbitrary external content directly. That endpoint validates public DNS resolution, pins requests to a checked public address, restricts redirects, applies its robots.txt checks, and enforces size and timing limits. When it returns an HTTP 200 HTML document, the browser analyzes the included meta elements and header fields. If the server rejects a disallowed host, times out, fails DNS validation, or returns a non-HTML resource, the interface shows an error. It must not translate any such failure into a healthy or indexable result.

When an Offline HTML Sample Is Appropriate

Pasted markup is useful when a development environment is private, when a live endpoint cannot reach the server, or when you want to compare two versions of a document. Choose offline mode, provide the intended URL for relative context, and paste source HTML. You may also provide an illustrative X-Robots-Tag value. The interface will clearly label the result as offline rather than verified HTTP data, because text pasted into a browser does not establish what any server currently delivers. This separation matters for audit reports: a developer may have corrected a file without deploying it, and a content-management system may transform headers or meta tags after source generation.

Robots.txt Is a Different Control

The restricted endpoint reads robots.txt before requesting permitted pages, but that safeguard is not a full robots.txt diagnosis. A Disallow rule describes which paths may be crawled by matching agents; it is not an instruction to remove a fetched page from an index. A crawler can discover a URL from links without being allowed to retrieve its page-level noindex directive. When planning changes, distinguish between allowing a crawler to fetch content so it can see noindex and blocking access to the resource entirely. The robots.txt generator can be used to draft rules only if it is among the verified published routes; never assume a generated rule has been deployed or honored.

Redirect Responses and Scope

A URL can return a permanent redirect to another path, host, or protocol. The destination response may be the one whose HTML is evaluated, but intermediate redirects can also carry meaningful headers. The inspector lists the observed hops and warns when redirects were involved rather than treating the starting and final URL as identical. A response scoped to Googlebot is not automatically identical to one scoped to Bingbot, and a request made with the inspection endpoint’s user agent may not reproduce a crawler-specific variant served to search engines. Compare the production response with appropriate server logs or a trusted crawler inspection feature when a directive behaves unexpectedly.

Exports and Review Workflow

Download the directive inventory as CSV when coordinating with developers and use the structured JSON report to retain the response history, header values, meta declarations, and warnings together. Before sharing a report, review the URL and any diagnostic strings for secrets embedded in query parameters. For a release checklist, record the exact page URL, environment, date, response code, and expected indexing policy. Then recheck a representative sample after deployment. The report is useful for identifying what the inspection endpoint observed on a particular request; it is not a substitute for search-engine indexing reports or a complete site crawl.

Limits of Automated Crawlability Judgments

No single meta or header inspection can prove that a page is eligible, discoverable, canonical, or likely to rank. The page could be blocked by authentication, inaccessible to some regions, dependent on scripts that a crawler does not execute as expected, or marked canonical to another URL. A sitemap entry is also not an indexing guarantee. The checker reports observable instructions and contradictory combinations, then leaves broader conclusions to a review involving live rendering, response codes, canonical tags, internal paths, and crawler-specific tools. This narrower claim is more accurate than presenting an overall pass/fail score that would conceal unknown conditions.

Frequently Asked Questions

Conclusion

A trustworthy robots audit preserves the distinction between an observed header, a document meta tag, a redirect response, and an actual crawler decision. Recheck the deployed response after changes, particularly when caching or infrastructure can modify headers.