These controls solve different problems. A robots.txt rule governs whether a crawler may request a path. Meta robots and X-Robots-Tag govern indexing, link following, and preview behavior after the crawler can retrieve the instruction.
Choose the control that matches the task
| Control | Delivered in | Use it for |
|---|---|---|
robots.txt | Site-root text file | Managing crawler access to paths |
| Meta robots | HTML <head> | HTML-page indexing and preview policy |
X-Robots-Tag | HTTP response header | Non-HTML files or server-managed indexing policy |
Do not use robots.txt as an indexing switch
Blocking a URL in robots.txt does not provide a noindex directive. It can also prevent a crawler from seeing a noindex instruction in the page or response header. If removal from search is the goal, the crawler must ordinarily be able to retrieve the applicable indexing directive.
Use meta robots for page-level HTML policy
Place one clear robots policy in the page head when an HTML page needs explicit instructions such as noindex, nofollow, or preview limits. Check generic and crawler-specific tags for contradictions and duplicates.
Use X-Robots-Tag when the policy belongs in the response
The HTTP header is appropriate for PDFs, images, and other resources without an HTML head. It can also centralize policy at the web server or CDN. Inspect the live response: pasted HTML alone cannot reveal this header.
Diagnose controls in a reliable order
- Confirm that the live URL returns the expected response.
- Check robots.txt for crawler and path-specific access rules.
- Inspect rendered HTML for generic and crawler-specific meta robots tags.
- Inspect HTTP headers for X-Robots-Tag directives.
- Compare the controls and remove unintended conflicts.
- Test the production URL with the relevant search-provider tools.
Check a meta robots policy Check robots.txt rules
Robots controls FAQ
Is robots.txt the same as noindex?
No. robots.txt controls crawling. A noindex directive in a meta robots tag or X-Robots-Tag header requests exclusion from an index after the crawler can retrieve the directive.
When should I use a meta robots tag?
Use it for HTML pages when the indexing or preview policy belongs in the page head and the crawler can access the page.
When is X-Robots-Tag useful?
Use the HTTP header for non-HTML resources such as PDFs or when indexing policy is controlled at the server or CDN layer.
Can a robots.txt block hide a noindex directive?
Yes. If crawling is blocked, a crawler may be unable to retrieve the page or response header containing noindex. Diagnose crawl access and indexing policy separately.