robots.txt generator
Enter the paths search crawlers can visit or avoid, then generate a robots.txt file with a sitemap reference in your browser.
robots.txt settings
Enter one path per line. Paths must start with /.
Organize search crawler rules
A robots.txt file tells search crawlers which site paths they should use as crawl hints. It is useful for keeping admin pages, search result pages, temporary paths, and low-value filtered URLs out of regular crawling.
Enter your site URL to create a default sitemap entry, then add blocked and allowed paths one per line. The generated text can be copied or downloaded as robots.txt and placed at the root of your site.
robots.txt is not a security feature. It is a public instruction file for crawlers, not a way to hide private pages. Sensitive content still needs login protection, permissions, or server-side access control.
Related tools
Continue with tools that fit the same task flow.