Control crawler accessTell search engine crawlers which areas they may or may not crawl.
Important safety note
Robots.txt controls crawler access, but it does not securely hide private content. For sensitive content, use authentication or proper access controls.
Reduce configuration errorsCreate correctly formatted rules without writing robots.txt syntax manually.
Add sitemap directives easilyInclude your XML sitemap so crawlers can discover important URLs more efficiently.
Frequently asked questions
What is a robots.txt file?+
It is a plain-text file at your site root that provides crawl guidance to compatible web crawlers.
Where should robots.txt be placed?+
Place it at the root of the host you want to control, such as https://example.com/robots.txt.
Can robots.txt block a page from Google indexing?+
Not reliably. It can prevent crawling, but a URL may still appear in search results. Use noindex and appropriate access controls when needed.
What is the difference between Allow and Disallow?+
Disallow identifies paths you ask a crawler not to access. Allow can be used to make an exception within a broader blocked path.
Should I add my sitemap to robots.txt?+
It is a useful, widely supported way to help crawlers discover your published XML sitemap.
Does Google support crawl-delay?+
Google does not support the crawl-delay directive. It is included here only because some other crawlers may recognize it.
Can I block all search engines with robots.txt?+
You can request that compatible crawlers do not crawl your site, but robots.txt is not a security control and cannot guarantee privacy or deindexing.
Related SEO Tools
XML Sitemap GeneratorCreate an XML sitemap from your page URLs.
›
XML Sitemap CheckerValidate sitemap structure and URL entries.
›Website SEO AuditFind technical SEO issues and opportunities.
›
Canonical Tag CheckerCheck canonical tags and indexing conflicts.
›