Robots.txt Checker
SiteTend Robots
Find malformed, conflicting, duplicated, or high-risk crawler rules before publishing a robots.txt file.
Paste robots.txt
Add the exact plain-text file content.
Review the rules
Start with indexing risks and invalid or conflicting directives.
Your analysis will appear here
Paste the file content and select Analyze robots.txt.
Help
Express crawler rules clearly
Assign rules to a User-agent group, use one directive per line, and review broad Disallow patterns carefully. Robots.txt controls crawling rather than guaranteed search indexing.
Questions
SiteTend Robots FAQ
What does SiteTend Robots check?
It checks syntax, crawler groups, known directives, duplicates, Allow and Disallow conflicts, empty rules, crawl delays, sitemaps, wildcards, broad indexing blocks, and recognized AI crawler groups.
Does robots.txt remove pages from search results?
No. Robots.txt controls crawling requests. A blocked URL may still be known or indexed through other signals.
Does the checker fetch my live robots.txt file?
No. This version analyzes only the file content you paste.
Does SiteTend store pasted content?
No. Analysis runs in your browser, and pasted robots.txt content is excluded from analytics.
Apply the result
From finding to verified fix
Interpret the finding
Treat blocking findings by crawler and path. A syntactically valid rule can still prevent important discovery.
Common causes to inspect
- Rules copied from staging
- Over-broad wildcards
- Conflicting user-agent groups or missing sitemap discovery
Practical next steps
- Identify the affected crawler
- Test representative important URLs
- Publish carefully and verify with the crawler's own tools
What this check cannot prove
The checker does not fetch the live file or prove that a blocked URL is indexed or removed from search.