Does naming the crawlers change what gets fetched?

No. User-Agent: * with Allow: / already admits every one of them. The named groups are there to be read: by the crawler, which some operators document as checking for its own name, and by anyone auditing whether the site has taken a position. Their absence reads as not having thought about it, and it is check three of the nine.

Should I block GPTBot to keep my pages out of training?

That is your call, and it is a legitimate one. Block GPTBot, ClaudeBot and Google-Extended if you want to, and keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and the three -User agents open, because those are the ones that fetch a page to answer a customer. The standard counts any named block against a site, since a crawler you shut out cannot quote you; a considered split is still worth more than a blanket allow nobody wrote.

Will robots.txt keep a crawler out?

It is a request, not a control. Well-behaved crawlers honour it and the named operators above document that theirs do. Anything that ignores it still gets the page. Nothing confidential should ever be relying on this file, on any site.

Where does the llms.txt pointer go?

As a comment on its own line after the Sitemap line: a hash, the word llms.txt, a colon, and the full address. There is no robots directive for it, and a comment is what a crawler that knows to look for it reads.

How do I know it worked?

Load yoursite.com/robots.txt in a private window and read it: plain text, the named groups present, the sitemap and the pointer at the foot. Then run the address through the checker on the AI-friendly page, which reads the live file and scores checks one to five from it.

AI-friendly websites · robots.txt

Let the assistants in, by name.

The crawlers behind ChatGPT, Claude, Perplexity, Gemini and the rest already get in through your wildcard. Naming them states a position, declaring the sitemap hands them the page list, and one comment tells them where your llms.txt is. That is five of the nine checks, and in a study of 281 trades websites, nine sites named the crawlers and none carried the pointer. Here is the block.

The block Check my robots.txt

robots.txt: the wildcard, then every crawler admitted by name.

The last line points at the llms.txt, which is where the answer lives.

Copy this

The whole file, for a site with nothing to hide.

Replace yoursite.com in the three places it appears. If your site has pages that should stay out of an index (an admin, a private area), add a Disallow: line for each under the wildcard group and under the named group, because a named group inherits nothing.

The list is OpenAI, Anthropic, Perplexity, xAI, Google, Apple, Amazon and Meta: every token from the operator's own documentation, except xAI's three, which xAI has never documented and which are the strings the crawler directories report. It is the same list our own robots.txt is generated from.

Ours, as served What goes at the address it points to

robots.txt

User-Agent: *Allow: / # The assistants that read this site to answer a question are welcome.# The plain-text account written for them is at https://yoursite.com/llms.txt.User-Agent: GPTBotUser-Agent: OAI-SearchBotUser-Agent: ChatGPT-UserUser-Agent: ClaudeBotUser-Agent: Claude-SearchBotUser-Agent: Claude-UserUser-Agent: PerplexityBotUser-Agent: Perplexity-UserUser-Agent: GrokBotUser-Agent: xAI-GrokUser-Agent: Grok-DeepSearchUser-Agent: Google-ExtendedUser-Agent: Applebot-ExtendedUser-Agent: AmazonbotUser-Agent: meta-externalagentAllow: / Sitemap: https://yoursite.com/sitemap.xml# llms.txt: https://yoursite.com/llms.txt

Line by line

Who each name is, and what it fetches.

Three kinds of crawler share the list. Training crawlers read the web to build a model. Search crawlers build the index an assistant answers from. User agents fetch one page because a person has just asked a question about it, which is the customer. The distinction matters if you want to admit some and not others.

Token	Operator	What it is for
GPTBot	OpenAI	OpenAI's crawler for training data.
OAI-SearchBot	OpenAI	OpenAI's crawler for ChatGPT search results.
ChatGPT-User	OpenAI	Fetches a page because a ChatGPT user asked about it.
ClaudeBot	Anthropic	Anthropic's crawler for training data.
Claude-SearchBot	Anthropic	Anthropic's crawler for search results.
Claude-User	Anthropic	Fetches a page because a Claude user asked about it.
PerplexityBot	Perplexity	Perplexity's index crawler.
Perplexity-User	Perplexity	Fetches a page because a Perplexity user asked about it.
GrokBotreported	xAI	xAI's crawler, as reported by crawler directories.
xAI-Grokreported	xAI	xAI's crawler, as reported by crawler directories.
Grok-DeepSearchreported	xAI	xAI's research crawler, as reported by crawler directories.
Google-Extended	Google	Google's token for Gemini and AI Overviews grounding. Blocking it does not affect Google Search.
Applebot-Extended	Apple	Apple's token for training Apple Intelligence. Blocking it does not affect Applebot for Siri and Spotlight.
Amazonbot	Amazon	Amazon's crawler, which feeds Alexa's answers.
meta-externalagent	Meta	Meta's crawler for training and its AI products.

8 operators, 15 tokens, as documented in September 2026. “Reported” marks a token the operator has not documented and the crawler directories list.

What we found

Four mistakes, from 281 real robots.txt files.

The block above covers checks one to five of the nine. These are the ways the 281 trades websites in the study failed them.

Blocking one of them by name

Three of the 281 did. One shut out ChatGPT-User and Claude-User, the two agents that fetch a page because a customer has just asked about the business. That is a customer standing at the door. If the worry is training data, block GPTBot and Google-Extended and leave the search and user agents open; that is a considered position and reads as one.

A robots.txt that is not a text file

Twenty-eight of the 281 had no usable one: eleven answered 404 with a full HTML error page, seven answered 200 with an empty body, and the rest refused or never connected. A crawler handed a web page where the rules should be reads it as no rules at all, which is a different thing from a policy.

Never mentioning the file

Not one of the 281 sites pointed at its llms.txt from robots.txt, including every site that had gone to the trouble of publishing one. There is no directive for it, so it goes in as a comment on its own line after the sitemap, the way the block above does.

Forgetting the named groups inherit nothing

A named group does not fall back to the wildcard's rules. If your wildcard group disallows /admin, every named group has to repeat the disallow, or the named crawler is admitted to the very pages the wildcard kept it out of.

robots.txt answers, in plain text

The assistants are not blocked

They are admitted by name

The sitemap is declared

robots.txt points at the llms.txt

View the study

Where it goes

Where the file lives on each platform.

The file is served at the root of the domain. Where you edit it depends on what built the site, and on two of the common platforms it cannot be edited at all.

WordPress: Yoast SEO and Rank Math both have a robots.txt editor under their Tools or General settings. Paste the block, save, then load yoursite.com/robots.txt in a private window to confirm it is the file being served and not a plugin's default.
Wix: Settings, then SEO, then robots.txt editor. Wix writes its own sitemap line, so add the named groups above it and the llms.txt comment below it.
Duda: Site settings, SEO, robots.txt. The builder generates an llms.txt of its own that lists your pages; the comment should point at a file you have written, not that one.
Squarespace: The file itself cannot be edited. There is a setting for whether AI crawlers may read the site; turn it on, and know that the named groups, the pointer and check three are out of reach on this platform.
A site of ours: Already done, from the same list this page is generated from. The named groups, the sitemap and the pointer are emitted by a route, the build fails without them, and the live file is read after every deploy.

Straight answers

The questions this page gets.

Does naming the crawlers change what gets fetched?: No. User-Agent: * with Allow: / already admits every one of them. The named groups are there to be read: by the crawler, which some operators document as checking for its own name, and by anyone auditing whether the site has taken a position. Their absence reads as not having thought about it, and it is check three of the nine.
Should I block GPTBot to keep my pages out of training?: That is your call, and it is a legitimate one. Block GPTBot, ClaudeBot and Google-Extended if you want to, and keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and the three -User agents open, because those are the ones that fetch a page to answer a customer. The standard counts any named block against a site, since a crawler you shut out cannot quote you; a considered split is still worth more than a blanket allow nobody wrote.
Will robots.txt keep a crawler out?: It is a request, not a control. Well-behaved crawlers honour it and the named operators above document that theirs do. Anything that ignores it still gets the page. Nothing confidential should ever be relying on this file, on any site.
Where does the llms.txt pointer go?: As a comment on its own line after the Sitemap line: a hash, the word llms.txt, a colon, and the full address. There is no robots directive for it, and a comment is what a crawler that knows to look for it reads.
How do I know it worked?: Load yoursite.com/robots.txt in a private window and read it: plain text, the named groups present, the sitemap and the pointer at the foot. Then run the address through the checker on the AI-friendly page, which reads the live file and scores checks one to five from it.

Paste it, then check it.

The free checker reads your live robots.txt and scores checks one to five from it, then goes on to the file it should be pointing at. It takes about ten seconds and stores nothing.

Start your project See the proof first

Or skip the form entirely. hello@pinelandtech.com reaches a person, not a queue.

Score my site