Free tool
robots.txt generator
Build a correct robots.txt in seconds — pick a policy, add the paths you want blocked, point crawlers at your sitemap, and copy the result.
Crawl rules
robots.txt
Blocking a path stops crawling, not indexing. To de-index a page, allow it and add a noindex meta tag.
FAQ
robots.txt questions, answered
What does robots.txt do?
robots.txt tells search-engine crawlers which parts of your site they may or may not request. It lives at the root of your domain (yourdomain.com/robots.txt) and is the first thing most crawlers read.
Does robots.txt stop a page from being indexed?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap: line helps crawlers discover all your URLs. You can list more than one sitemap.
Is anything sent to a server?
No. The file is generated in your browser — copy it and upload it to your own site.
Technical SEO, handled for you.
SERP Agent audits robots.txt, sitemaps, indexing and 30+ other technical signals — then fixes what’s wrong, automatically.