robots.txt generator
Create your site's robots.txt file: allow or block search and AI bots and private folders, and point to your sitemap.
How to userobots.txt generator
Available in: 
robots.txt generator: what it is for
Build your site's robots.txt file without memorizing the syntax. Choose whether bots can crawl the site, list the folders and pages to keep out, the exceptions and the sitemap address.
Check the bots you want to block, such as AI bots (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) and SEO tool bots. Copy or download the ready file.
How to use
- Choose the general rule and enter the sitemap.
- List blocked folders and check the bots.
- Copy or download robots.txt.
When to use it
- Creating robots.txt for a new site.
- Preventing AI bots from using the site's content.
- Blocking admin areas and search pages.
Frequently asked questions
Where do I put robots.txt?
At the root of the domain, available at yoursite.com/robots.txt. Each subdomain needs its own file.
Does blocking in robots.txt remove a page from Google?
Not necessarily. robots.txt prevents crawling, but a blocked page can still appear if it has links. To remove it from the index, use the noindex meta tag and leave the page crawlable.
How do I block AI bots?
Check the AI bots in the list, such as GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended and CCBot (Common Crawl). They state that they respect robots.txt.