Generate robots.txt files with visual editor. Block bots, set crawl rules, add sitemaps — with templates for common setups.
About This Tool
The robots.txt file tells search engine crawlers which pages they can and cannot access on your website. It sits in your site's root directory and is the first file any well-behaved crawler checks before indexing your content.
A misconfigured robots.txt can block Google from indexing your entire site, or accidentally expose admin pages you wanted hidden. This generator helps you build a correct robots.txt with a visual editor — add rules per bot, set allow/disallow paths, include sitemap URLs, and use templates for common setups.
The generated file follows the Robots Exclusion Protocol standard supported by Google, Bing, Yandex, and all major search engines.
How to Use
1. Start with a template or build from scratch
2. Add user-agent groups — each group targets a specific bot (or all bots with *)
3. Add Allow and Disallow rules for each group
4. Optionally set Crawl-delay for bots that support it
5. Add your sitemap URL(s) at the bottom
6. Copy the generated robots.txt and upload it to your site's root directory
Frequently Asked Questions
Where do I put the robots.txt file?
The robots.txt file must be placed in the root directory of your website, accessible at https://yourdomain.com/robots.txt. It won't work in subdirectories.
Does robots.txt block pages from appearing in Google?
Not exactly. Robots.txt blocks crawling, not indexing. If other sites link to a page you've disallowed, Google may still show it in search results (without a snippet). To fully block indexing, use a 'noindex' meta tag instead.
What does 'User-agent: *' mean?
The asterisk (*) is a wildcard that matches all crawlers. Rules under 'User-agent: *' apply to every bot that doesn't have its own specific section in the file.
Should I block AI crawlers like GPTBot?
That depends on your preference. If you don't want your content used for AI training, you can add 'User-agent: GPTBot' with 'Disallow: /' to block OpenAI's crawler. Similar rules exist for other AI bots like Google-Extended and CCBot.
What is Crawl-delay?
Crawl-delay tells bots to wait a specified number of seconds between requests. It's supported by Bing and Yandex but ignored by Google. Use it if aggressive crawling is overloading your server.
Can I use wildcards in robots.txt paths?
Google and Bing support limited wildcards: * matches any sequence of characters, and $ marks the end of a URL. For example, 'Disallow: /*.pdf$' blocks all PDF files. Not all bots support these extensions.