Keep rules small and explainable
Prefer directory-oriented rules over long duplicated URL lists, and review any rule that could block CSS, JavaScript, or important pages.
CRAWL CONTROL
Robots.txt Checker
Audit crawl access and keep discovery reliable.
Score crawler rules and sitemap declarations.
Start nowNo account required
Live
01Score
02Conflicts
03Sitemaps
Webmaster Technical SEO and AI Discoverability Guide
Learn how robots.txt, sitemap.xml, and llms.txt work together.
A domain checks /robots.txt at its root; a full file URL checks that exact location.
Scoring Notes
- Required: HTTP 200 +25, valid directive syntax +20, recognized standard directives +15.
- Important: no matching Allow/Disallow conflicts +10; declared sitemaps are valid and reachable +15.
- Enhanced: specific user-agents ordered first +5, no empty Disallow +5, file within 500 KB +5.
About robots.txt: manage crawling before optimizing discovery
A robots.txt file is placed at a domain root to tell compliant crawlers which paths they may request. It is useful for managing crawl access and resource consumption, but it is not a substitute for noindex controls, authentication, or content removal.
A small configuration error may block important resources or hide a sitemap discovery path. This checker evaluates file availability, directive syntax, path conflicts, declared sitemap validity, and the 500 KiB processing limit. Crawl-delay is recognized for auditing, but it is not among the fields supported by Google in robots.txt.
Robots.txt Best Practices
Declare a working sitemap
Add a complete sitemap URL and make sure it is accessible and valid XML so crawlers have a reliable discovery entry point.
Do not use robots as secrecy
Protect sensitive content with authentication and use appropriate indexing controls for accessible pages that should not be indexed.
Stay within the size limit
Google processes the first 500 KiB of robots.txt. Consolidate rules before the file approaches that limit.
Frequently Asked Questions
Can robots.txt prevent a page from appearing in search results?
Not reliably. Robots.txt manages crawling; a URL found through other links may still appear without a useful snippet.
Why declare a sitemap in robots.txt?
A full sitemap URL provides supporting search engines with a consistent discovery location and makes technical audits easier.
What does an empty Disallow directive mean?
An empty Disallow directive does not block a path, effectively permitting crawling for that rule and making misconfiguration easy.
Does Crawl-delay work for Googlebot?
Google does not list Crawl-delay among its supported robots.txt fields. Use Google-provided crawl management options when needed.