{"title":"AI Search Crawlers and Training Controls: What Publishers Can Choose","url":"https://outwrite.ai/research/ai-search-crawlers-and-training-controls","type":"Research review","modified":"2026-11-04T15:55:47.508467+00:00","methodology":"We compared the documented controls of three providers against the Robots Exclusion Protocol and Google’s request-verification guidance on November 4, 2026. We classified each control by use and compared it with what a publisher could check independently. The worked cases are proposed decision exercises. We did not sample crawler traffic or measure citation rates.","disclosure":"outwrite.ai publishes content and sells visibility tracking, giving it a commercial interest in how teams measure visibility. This review covers first-party documentation and reports no original measurements of provider compliance.","sources":[{"id":"1","title":"OpenAI crawler overview","publisher":"OpenAI","url":"https://developers.openai.com/api/docs/bots","finding":"OAI-SearchBot serves ChatGPT search. GPTBot collects content potentially used to train OpenAI foundation models. The controls are independent. ChatGPT-User is user-directed and robots rules may not apply. Navigational links are a search opt-out exception.","limitation":"Describes OpenAI controls, not citation frequency, retroactive removal, or all AI services.","verified":true,"sample":"Official documentation; no measured page or prompt sample.","period":"Version inspected November 4, 2026.","checked":"2026-11-04"},{"id":"2","title":"Anthropic crawler controls","publisher":"Anthropic","url":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","finding":"ClaudeBot concerns potential training use of future material, Claude-SearchBot concerns search, and Claude-User handles user-directed retrieval; Anthropic documents separate opt-outs for its three agents. Anthropic states that disabling Claude-User prevents its system from retrieving this content in response to user queries.","limitation":"Do not assume Anthropic's user-request policy applies to ChatGPT-User.","verified":true,"sample":"Official documentation; no measured page or prompt sample.","period":"April 7, 2026 documentation.","checked":"2026-11-04"},{"id":"3","title":"Google’s common crawlers and Google-Extended","publisher":"Google","url":"https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended","finding":"Google-Extended is a robots token without a distinct HTTP user agent. It covers specified future Gemini training and grounding uses. It does not affect Google Search inclusion or ranking. Covers training future generations of Gemini models powering Gemini Apps and Vertex AI API for Gemini, and grounding in Gemini Apps and Grounding with Google Search on Vertex AI.","limitation":"This token couples certain uses. Do not generalize its scope to every Google AI feature.","verified":true,"sample":"Official documentation; no measured page or prompt sample.","period":"October 30, 2026 documentation.","checked":"2026-11-04"},{"id":"4","title":"Robots Exclusion Protocol","publisher":"IETF","url":"https://www.rfc-editor.org/rfc/rfc9309.html","finding":"Matching groups combine for the same product token. The wildcard group is used when no specific matching group exists. Matching path specificity decides which rule applies. The file is public and is not authorization. Section 2.4 permits caching and recommends against using a cached file beyond 24 hours unless the file is unreachable.","limitation":"A standard defines compliant behavior, not evidence that any particular requester complied.","verified":true,"sample":"Official documentation; no measured page or prompt sample.","period":"September 2022 standard.","checked":"2026-11-04"},{"id":"5","title":"Introduction to robots.txt","publisher":"Google Search Central","url":"https://developers.google.com/search/docs/crawling-indexing/robots/intro","finding":"Blocking a crawl does not necessarily stop a URL appearing in search. Sensitive resources require an actual access control.","limitation":"The guidance describes Google Search and general risks; it is not a contract about other products.","verified":true,"sample":"Official documentation; no measured page or prompt sample.","period":"Version inspected November 4, 2026.","checked":"2026-11-04"},{"id":"6","title":"Verify requests from Google crawlers and fetchers","publisher":"Google","url":"https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests","finding":"Google documents forward-confirmed reverse DNS or its published IP ranges for verifying request origin.","limitation":"Origin verification identifies the requester; it is not proof of a downstream answer, training event, or attribution.","verified":true,"sample":"Official documentation; no measured page or prompt sample.","period":"Version inspected November 4, 2026.","checked":"2026-11-04"},{"id":"7","title":"What is Googlebot?","publisher":"Google Search Central","url":"https://developers.google.com/search/docs/crawling-indexing/googlebot","verified":true,"sample":"Provider documentation, no measured sample.","period":"Version inspected November 4, 2026.","checked":"2026-11-04","finding":"Googlebot denotes Google Search crawlers. A request header can be spoofed. Google refers site owners to its request verification methods.","limitation":"Identifying a Search crawler does not establish ranking or inclusion in an AI answer."}],"chart":null}