For transactional email
Consider SMTP2GO for website-generated messages, form notifications, account notices, and other outbound system email.
Explore SMTP2GOAvailableFree browser-local analysis
Robots.txt Checker
SiteTend Robots
Find malformed, conflicting, duplicated, or high-risk crawler rules before publishing a robots.txt file.
Your input stays private.Robots.txt content is analyzed in your browser.
Paste robots.txt
Add the exact plain-text file content.
Review the rules
Start with indexing risks and invalid or conflicting directives.
Your analysis will appear here
Paste the file content and select Analyze robots.txt.
Help
Express crawler rules clearly
Assign rules to a User-agent group, use one directive per line, and review broad Disallow patterns carefully. Robots.txt controls crawling rather than guaranteed search indexing.
Questions
SiteTend Robots FAQ
What does SiteTend Robots check?
It checks syntax, crawler groups, known directives, duplicates, Allow and Disallow conflicts, empty rules, crawl delays, sitemaps, wildcards, broad indexing blocks, and recognized AI crawler groups.
Does robots.txt remove pages from search results?
No. Robots.txt controls crawling requests. A blocked URL may still be known or indexed through other signals.
Does the checker fetch my live robots.txt file?
No. This version analyzes only the file content you paste.
Does SiteTend store pasted content?
No. Analysis runs in your browser, and pasted robots.txt content is excluded from analytics.
Apply the result
From finding to verified fix
Interpret the finding
Treat blocking findings by crawler and path. A syntactically valid rule can still prevent important discovery.
Common causes to inspect
- Rules copied from staging
- Over-broad wildcards
- Conflicting user-agent groups or missing sitemap discovery
Practical next steps
- Identify the affected crawler
- Test representative important URLs
- Publish carefully and verify with the crawler's own tools
What this check cannot prove
The checker does not fetch the live file or prove that a blocked URL is indexed or removed from search.
Recommended resources
Useful next steps for operating a website
For measuring business inquiries
Use WhatConverts to measure which marketing sources produce calls, forms, and leads. It measures outcomes; it does not fix technical website issues.
Explore WhatConvertsFor personal-data privacy
Consider Optery if exposed personal information or data-broker listings are a concern. It is a separate privacy resource, not an SEO or website-security scanner.
Explore OpterySiteTend may earn a commission if you purchase through these links, at no additional cost to you. These services are optional and separate from SiteTend's free checks.