Create a robots.txt without breaking your SEO
Pick your platform, choose which crawlers to allow, and get a ready-to-upload file. We check it live, so you see what Google and each AI bot will actually do.
Platform
Crawlers
Your robots.txt
# robots.txt generated by freelocalseoaudit.com/tools/robots-txt-generator
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/
Sitemap: https://yourbusiness.com/sitemap.xml
What crawlers will do with this file
- GooglebotPartly allowed
- BingbotPartly allowed
- OAI-SearchBotPartly allowed
- PerplexityBotPartly allowed
- Claude-SearchBotPartly allowed
- GPTBotPartly allowed
- ClaudeBotPartly allowed
- Google-ExtendedPartly allowed
“Partly allowed” means the crawler can read your site except the Disallow paths above, which is normal. Uploaded it already? Check your live robots.txt →
Common robots.txt mistakes
Blocking the whole site by accident
A single “Disallow: /” under “User-agent: *” removes you from Google. It's common after a site leaves staging.
Blocking CSS and JavaScript
Disallowing /wp-content/ or /wp-includes/ stops Google rendering your pages properly, which can hurt rankings.
Using robots.txt to hide pages
Blocked pages can still be indexed if other sites link to them. Use a noindex tag to keep a page out of Google.
Robots.txt questions
A plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which parts of the site they may visit. Google, Bing and the major AI crawlers follow it.
To your site's root folder, so it opens at yoursite.com/robots.txt. On WordPress, use your host's File Manager or FTP, or the robots.txt editor in Yoast SEO or Rank Math. On Shopify, edit the robots.txt.liquid theme template.
Blocking AI training crawlers (GPTBot, ClaudeBot, Google-Extended) is a legitimate choice and doesn't affect Google or AI search. Blocking AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) stops those products from citing and linking to you, which most local businesses don't want.
A crawler named in its own group ignores the “User-agent: *” rules. Blocked groups only need “Disallow: /”, so nothing else has to be repeated there.
No. It stops crawling, not indexing. To remove a page from search results, let Google crawl it and add a noindex meta tag.
Uploaded it? Check it.
See which AI crawlers your live robots.txt allows, or run the full free audit, which checks robots.txt alongside your redirects, schema and more.
Rated 4.8/5 from 126 ratings