Some AI crawlers collect content to train AI models. Others fetch your pages so your business can appear in AI search answers. They are separate, and you can set each one on its own in a small text file called robots.txt. In our view, blocking everything usually costs a small business more than it gains. Our plain advice: allow the search ones, and decide on the training ones. Details and a copyable file below. This post is one part of our guide to an AI-ready website. Crawlers being able to get in is check number one on that list.
If you also want your pages to carry honest structured data, see schema markup for small business. Once your file is set, you can test what the AI tools say about you: see How to Check Whether AI Tools Mention Your Business.
What is robots.txt?
It is a text file at the root of your site, for example yourhotel.com/robots.txt. It tells crawlers which parts of your site they may visit. Google describes it as a tool mainly to manage crawler traffic (Google Search Central: introduction to robots.txt). It is not a way to hide a page. Google also says respectable crawlers follow it, but it cannot force any crawler to obey, and other crawlers might not.
What do the main AI crawlers do?
Each company publishes its own list. Here is what their official pages say.
- GPTBot crawls content that may be used to train OpenAI's generative AI models. OpenAI says disallowing GPTBot indicates your content should not be used for that training.
- OAI-SearchBot is for search. OpenAI says it is used to show websites in ChatGPT's search features. Sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links.
- ChatGPT-User visits a page when a person asks ChatGPT something. OpenAI says robots.txt rules may not apply to it, because the user starts the action.
- OpenAI says each setting is independent. You can allow OAI-SearchBot and disallow GPTBot. OpenAI says search changes can take about 24 hours to take effect.
- Googlebot builds Google Search. Rules for Googlebot affect Search.
- Google-Extended is a separate token. Google says it controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect a site's inclusion in Google Search and is not a ranking signal.
- ClaudeBot collects web content that could contribute to training. Anthropic says restricting it signals that your future content should be left out of training data.
- Claude-User fetches pages when a person asks Claude a question. Anthropic says disabling it may reduce your visibility for user-directed web search.
- Claude-SearchBot helps improve search result quality. Anthropic says disabling it may reduce your visibility and accuracy in user search results.
- Anthropic says to use robots.txt and not to block IP addresses, because blocking IPs may stop it reading your robots.txt. It also says to repeat the rule for every subdomain.
What blocking does and does not change
- Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended) is how you say "don't use my content to train your models". OpenAI, Google and Anthropic say their training crawlers follow it.
- It does not change how your site is listed in Google Search. Google says Google-Extended has no effect there. Googlebot is what matters for Search, so don't block Googlebot.
- Blocking a search or user crawler (OAI-SearchBot, Claude-SearchBot, Claude-User) can make you less visible in that company's AI search. OpenAI and Anthropic say so on their own pages.
- Not confirmed: we found no official page that says blocking any crawler raises or lowers your Google ranking, or that it changes whether you are named in an AI answer beyond what is quoted above. Nobody can promise you a mention either way.
- robots.txt is a request, not a lock. For private content, use a password.
Who should allow, and who should block?
This is our view, not a rule from any vendor.
- Hotel, guesthouse or surf camp: guests ask AI tools where to stay. Allow the search and user crawlers. Whether to allow training crawlers is your choice. Your room pages and prices are public anyway. For more on being named in these answers, read Not Found in AI Search? What a B&B Owner Can Do.
- Restaurant or café: same. Menus, hours and your address are what you want repeated correctly. Allow search crawlers, and keep your menu as text, not only a PDF or image.
- Local shop or webshop: if your product photos and descriptions are your own work and you don't want them used for training, blocking the training crawlers is a reasonable choice. Keep search crawlers allowed so people can still find you.
- Not sure: allow the search ones, block nothing else yet, and revisit in a few months. You can change the file at any time.
A small robots.txt you can copy
This example allows search and user crawlers and asks the training crawlers to stay out. Delete the lines for crawlers you are happy to allow.
# Everyone else: allowed
User-agent: *
Disallow:
# AI training crawlers: please don't use my content for training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# AI search and user crawlers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xml
Replace the sitemap line with your own address. If you use WordPress, an SEO plugin or your host's settings page may let you edit this file. Ask your web person before changing it.
How to check your own file
- Open
yourwebsite.com/robots.txtin a browser. You should see plain text. If you get an error page, you have no file, and everything is allowed by default. - Look for
Disallow: /underUser-agent: *. That blocks every crawler, including Google. Remove it unless you want your site hidden. - Check the crawler names are spelled exactly as above.
- Open Google Search Console and look for the robots.txt report. Google shows the file it found, parsing errors and warnings, and has a recrawl request. Per Google's help page, the report is only available for Domain properties, or URL-prefix properties without a path.
- Check your host or CDN. In our experience, security settings can block bots before robots.txt is read, and you won't see that in the file.
- Wait. OpenAI says search changes can take about 24 hours; others may differ.
Common mistakes
- Leaving
User-agent: *withDisallow: /in place after a site rebuild. - Blocking Googlebot when you meant to block only Google-Extended.
- Using robots.txt to hide private pages. Google says a blocked URL can still be listed if other sites link to it. Use a password or a
noindextag instead. - Blocking IP addresses instead of using robots.txt. Anthropic says that may stop it reading your rules.
- Forgetting subdomains. Each one has its own robots.txt.
- Typing a crawler name wrong, so the rule does nothing.
- Expecting a guarantee. A blocked or allowed crawler does not promise a ranking or an AI mention.
Want help checking your site?
See our AI visibility service, or ask for a meeting.