AI Search Crawlers and Training Controls: What Publishers Can Choose

Can readers still find your manual?

OpenAI and Anthropic let publishers express separate preferences for search and model development 1 2. Those preferences concern automatic crawlers. Google has a different arrangement: Google-Extended combines certain Gemini training and grounding uses and does not affect Google Search ranking or inclusion 3. For a reader-triggered request, OpenAI says robots rules may not apply to ChatGPT-User 1. Anthropic describes a Claude-User opt-out 2. A robots.txt file keeps nothing private, and a request record alone gives us no basis for concluding that a page appeared in an answer or entered a training set. The real work is deciding which content gets which preference, putting anything sensitive behind sign-in, and reporting results only as far as the evidence goes. The sections below work through those decisions on a fictional documentation site with three areas: a public manual, an embargoed launch and licensed partner excerpts.

How this review was done

We inspected the sources on November 4, 2026. The Anthropic guide is dated April 7, 2026, the Google token list was updated October 30, 2026, and the IETF standard dates to September 2022. The inspection date is separate from those document dates. We used Python’s urllib.robotparser to check nine agent strings against two paths each (18 local checks). This exercises our example only. The groups use simple prefixes and contain no competing path rules. The test does not establish parser equivalence, provider compliance, user-request handling, or indexing. No live provider was involved, and we sampled neither server logs nor citations. The site, file and ticket below are invented. Statements about what a publisher should do are our recommendations, not provider guidance.

For every provider page your policy relies on, record the version and the date you read it. These pages change.

Agents and tokens by provider

The table tells you which names to look up. The policy depends on what each provider does with them.

OpenAI controls OAI-SearchBot and GPTBot independently, so a rule for one says nothing to the other. ChatGPT-User fetches pages on a user's behalf, and OpenAI says robots rules may not apply to it. OpenAI also lists navigational links as an exception to the search opt-out 1. Because of that, we would not treat disallowing OAI-SearchBot as a guarantee that a URL never appears.

Anthropic says that disabling Claude-User prevents its system from retrieving site content for a user query 2. It describes ClaudeBot in terms of potential training use of future material 2. Anthropic's approach to user requests is its own, so don't assume it applies to ChatGPT-User.

Google-Extended is a robots token with no HTTP user agent of its own. Google describes one preference covering both training future Gemini models and grounding answers in Gemini Apps and Grounding with Google Search on Vertex AI. Google Search inclusion and ranking are independent of this token 3. Don't stretch its scope to other Google AI features.

Before editing the file, write one sentence per agent stating the outcome you want: allow, disallow, or outside robots.txt control.

How the rules resolve

RFC 9309 defines the matching behavior behind the example below. Groups naming the same product token combine. The wildcard group applies only when no group names the crawler. Among the rules that match a path, the most specific path decides 4. The standard also notes that the file is public and is not authorization 4. Google adds that blocking a crawl does not necessarily keep a URL out of search, and that sensitive resources need real access control 5.

We recommend checking three consequences on every change:

• A wildcard restriction does nothing for an agent that has its own group, so repeat the restriction in that group.
• A stray second group for the same agent merges with the first and can change the outcome.
• Anyone can read the file, so listing a path tells people it exists.

Put anything whose exposure would cause harm behind authentication. Use robots.txt only to express preferences about content you are willing to publish.

A robots file for a public manual and licensed excerpts

Example Docs is a fictional company with three areas:

• /manual/: the public product manual.
• /partners/excerpts/: excerpts licensed from a partner, readable by signed-out visitors.
• /launch/: embargoed launch material, available only after sign-in.

The file below shows how Example Docs might state its preferences. It illustrates structure and is not a template.

# Hypothetical file for a fictional site. Illustration only.

User-agent: GPTBot
Disallow: /manual/
Disallow: /partners/

User-agent: ClaudeBot
Disallow: /manual/
Disallow: /partners/

User-agent: OAI-SearchBot
Disallow: /partners/

User-agent: Claude-SearchBot
Disallow: /partners/

User-agent: Google-Extended
Disallow: /manual/
Disallow: /partners/

User-agent: *
Disallow: /partners/

These are the local parser results for the exact file above. The rows for ChatGPT-User and Claude-User show only how the Python library interprets those strings. They cannot establish whether a reader-triggered service requests the page.

In this file the named groups all contain /partners/. For Googlebot the example uses the wildcard group. The local parser returned disallow on the sample partner URL and allow on the manual URL for Googlebot. These results describe the fixture, not a request from Google. Google separately warns that a crawl restriction alone cannot guarantee exclusion of a URL from Google Search 5.

Confirm that each named group carries every restriction you intended, not only the ones specific to that agent.

Three content classes, three decisions

Public manual. Example Docs wants people who ask an assistant about its product to reach accurate pages, so the two named search-crawler groups in the example contain no restriction on the manual. Training is a separate business decision. The publisher prioritizes limiting training use across all three providers. Its manual rules therefore disallow GPTBot, ClaudeBot and Google-Extended. For Google the publisher accepts the associated restriction on the covered grounding uses. This token does not affect Google Search inclusion or ranking 3. What matters is writing the reason down, because the file won't explain itself. Nothing reviewed here promises visibility or traffic in return for allowing search agents, so no one should promise it internally.

Embargoed launch. This is not a robots.txt problem. The pages sit behind sign-in and stay out of the file, which would otherwise advertise them. When the embargo lifts, the content moves into /manual/ and takes on the manual's policy. Treat that move as a planned reopening of the decision.

Licensed partner excerpt. Someone else's terms govern this content, so ask whoever owns the agreement what it requires. This article is not legal advice. Each group printed in the example disallows the partner path. The file illustrates preferences for automatic crawling only. It is still only a preference. As noted above, it may not bind every user-directed fetch, and Google says a blocked URL can still appear in Google Search 5. If the terms require restrictions on who can retrieve the excerpt, ask the owner to enforce those restrictions before sharing it.

Before choosing rules, classify every content area as public, public with conditions, or restricted.

Counting signed-out requests

You can measure your own controls more reliably than you can measure provider behavior. We recommend two counters, both limited to requests without a valid session.

For the fictional launch, we would group requests by status and then inspect the response to a synthetic request without a session. Suppose the server returns HTTP 200 with only the login form. We would record that separately from HTTP 200 with the announcement inside. The failure is returning the protected material to that reader. Check the response body using a harmless marker in a test document; keep private content out of routine logs.

For conditional paths such as /partners/excerpts/, count requests by declared user agent and status. These figures only show what requesters claimed to be. A user agent string is self-reported until you check where the request came from.

Neither counter says anything about training, citation or answers. Use the first counter to choose a response to inspect and the second to decide which claimed identities merit a closer look.

Define what opens a ticket. A signed-out disclosure on a gated path should page someone the same day. Unfamiliar declared agents on conditional paths can wait for a weekly review.

A worked incident ticket

This ticket is hypothetical, with placeholders instead of figures.

Ticket: DOCS-EXAMPLE
Summary: Signed-out requests declaring Googlebot under /partners/excerpts/
Window: [start] to [end]
Requests: [count], status [codes]
Live robots.txt: [fetched at], matches repository: [yes/no]
Origin check: [method], [result]
Classification: [unverified string / verified crawler / user-directed agent]
Conclusion: [one sentence]
Next step: [owner], [action], [date]

Work through it in order:

1. Fetch the live file as a signed-out client and compare it with the repository version and its deploy time. RFC 9309 permits caching 4. If the site changed its rules during the incident, retain the earlier version alongside the new one.
2. Google uses Googlebot for Search crawling 7. The example file has no Googlebot group. Check the wildcard group and whether a more specific path rule matches the excerpt URL.
3. Check where the requests came from, using the provider's current documented verification method. For Google agents, that is forward-confirmed reverse DNS or Google's published IP ranges. These identify the requester but are not proof of a downstream answer, training event or attribution 6. Record which method you used for which provider.

Then write a conclusion the evidence supports. "Requests declaring Googlebot reached a disallowed path; origin unverified" is fair. We would check the request origin 6, the requested path, and the policy version before assigning the ticket. Those details still cannot establish how a model used the content. If the excerpt's agreement cannot tolerate such requests, the fix is to gate the path, not to word the report more strongly.

No ticket closes without a classification and a named owner for the next step.

Reporting to different stakeholders

We recommend one shared vocabulary across reports:

• Requested: stated in robots.txt.
• Enforced: backed by access control.
• Observed: seen in logs.
• Verified: origin confirmed.

Each audience needs something different:

• Leadership and marketing need the per-agent preferences, what is enforced, and a plain statement that no visibility or traffic outcome is promised.
• Whoever owns the partner agreement needs dated facts: which rules cover the excerpt, which provider pages those rules rest on, and what was observed. That person judges whether it is adequate.
• The partner needs a short, honest account of what protects their excerpt and what does not.
• Engineering and security need the counters, any gated-path failures and the open tickets.

Name one owner per audience so reports don't drift into different claims.

Handover and when to reopen

The handover note should let someone new defend every line of the file. Include:

• the live file and its repository location
• the intended outcome per path and agent, with the reason
• each provider page relied on, with the date read
• the verification method for each provider
• counter definitions and alert rules
• open tickets
• the licensing contact
• the next review date

Reopen the decision when:

• a provider page is updated, or adds or renames an agent
• the embargo lifts
• a new content area appears
• the partner agreement changes
• a ticket closes with a verified origin on a disallowed path
• the business goal changes

Don't promise that a later change undoes earlier collection. Our review establishes nothing about removing previously collected content. Treat a signed-out disclosure on a gated path as an engineering defect.

At handover, put the next review date in the team calendar so it doesn't wait for the next incident.

Limitations

This review rests on seven documents, provider documentation plus the IETF standard, all read on one date. They describe intended behavior. A standard defines how compliant crawlers parse the file; it doesn't show that any requester complied 4. The local parser check tested the illustrated rules only. We did not measure live crawling, citations, traffic or training, and we did not cover other AI providers. The site, file and ticket are illustrations.

Treat any claim about outcomes, from your team or a vendor, as needing its own evidence before it goes in a report.