# OpenAI, Google, Anthropic, and Microsoft Back __usage_ai-training Standard for robots.txt
Summary: On October 30, 2026, Anthropic, OpenAI, Google, and Microsoft announced coordinated support for draft-kelley-aipref-training-preference-signal, an IETF Internet-Draft establishing the standardized __usage_ai-training policy token for robots.txt. The initiative allows website operators to express machine-readable opt-outs for generative AI model training through standard RFC 9309 directives, resolving years of fragmented bot-blocking overhead without disrupting traditional search indexing or real-time retrieval.
What Happened & Key Timeline
For three years, engineering teams managing web platforms faced an unsustainable operational burden: controlling whether generative AI models scraped their content for training required maintaining an ever-growing list of proprietary crawler user-agents. A publisher attempting to opt out of foundational training had to update robots.txt continuously to disallow GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Meta-ExternalAgent, Amazonbot, and dozens of new scrapers. Any omitted crawler resulted in unintended data ingestion.
On October 30, 2026, the industry moved to resolve this fragmentation. Kevin Kelley, technical and standards lead at Anthropic, published an Internet-Draft through the Internet Engineering Task Force (IETF) titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal). Submitted under the IETF AI Preferences (AIPREF) Working Group, the specification introduces a shared policy token—__usage_ai-training—designed to indicate machine-readable training opt-outs directly within standard robots.txt files.
Simultaneously, four frontier AI lab operators—Anthropic, OpenAI, Google, and Microsoft—issued coordinated commitments pledging to honor the token across their scraping, data acquisition, and model training pipelines.
The transition timeline outlines phased implementation across late 2026 and early 2027:
- October 30, 2026: Anthropic enabled immediate support for
__usage_ai-trainingacross its ingestion infrastructure upon specification publication. - Q4 2026: OpenAI confirmed development to parse and honor the token across automated training datasets over coming release cycles.
- February 2027: Microsoft committed to production enforcement across Bing-associated AI training pipelines and Azure AI scrapers.
- Early 2027: Google scheduled phased support, mapping the unified token alongside its existing Google-Extended crawler governance controls.
Technical & Architectural Impact
The primary architectural achievement of draft-kelley-aipref-training-preference-signal lies in its compliance with RFC 9309 (the Robots Exclusion Protocol standard). Rather than proposing an entirely new protocol or introducing bespoke file formats, the specification operates within established internet infrastructure.
RFC 9309 Syntax and Namespace Design
The standard uses the User-Agent record line in robots.txt to declare a policy token rather than a specific physical user-agent. The leading double underscore (__) establishes an explicit namespace convention, preventing collisions with crawler identifiers while signaling that the record represents a cross-vendor policy declaration:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Because the syntax conforms to RFC 9309 path matching, existing robots.txt parsers require zero structural modifications. Legacy crawlers and search indexers that do not implement the AIPREF standard treat __usage_ai-training as an unrecognized crawler token and ignore the block without throwing syntax errors.
Decoupling Training Ingestion from Search Discovery
Historically, webmasters faced a false choice: allow all crawlers to ensure discoverability in search engines, or block web crawlers aggressively and risk severe organic traffic destruction. While tokens like Google-Extended allowed publishers to differentiate Google Search indexing from Gemini training, the lack of industry-wide standardization created confusion.
Under the AIPREF taxonomy, the __usage_ai-training directive applies strictly to model training—defined formally as using an asset to modify the learned parameters of a generative AI foundation model. It explicitly separates data harvesting for model pre-training from:
1. Search Indexing: Crawling and caching documents to present links, titles, and snippets in web search engine indexes.
2. Real-Time Retrieval & Grounding: Live queries where an agent fetches an authorized web page at inference time to answer an end-user prompt with source citations.
This separation ensures that an enterprise can disallow foundation model training on its technical articles while retaining complete visibility in algorithmic web search.
Section 4.5 Precedence and Granular Overrides
A central question for systems architects is how the generic policy token interacts with crawler-specific directives. Section 4.5 of the draft specification establishes clear precedence rules:
- Specificity Wins: More specific expressions take precedence over general expressions. If an enterprise specifies
Disallow: /underUser-Agent: __usage_ai-trainingbut includesUser-Agent: ClaudeBotwithAllow: /, Anthropic's crawler respects the explicit ClaudeBot rule. - Proprietary Mappings: Crawler operators are permitted to treat their proprietary tokens as more specific than the generic token.
- Bilateral Overrides: Private commercial contracts, API access licenses, and data syndication agreements explicitly override
robots.txtdirectives, preserving structured enterprise partnerships.
What This Means for Engineering Teams & Enterprises
For platform engineers, Site Reliability Engineers (SREs), and enterprise web architects, the emergence of a standardized token fundamentally alters how web governance is managed.
Edge Caching and Dynamic robots.txt Generation
Modern cloud platforms operating at high scale frequently generate robots.txt through Content Delivery Network (CDN) edge workers like Cloudflare Workers, Fastly Compute, or AWS CloudFront Functions. Previously, edge workers maintained expansive regular expression dictionaries to detect and block dozens of distinct AI bots by header string. This pattern expanded memory footprints and required continuous pipeline updates whenever a vendor launched a new scraper.
With __usage_ai-training, engineering teams can replace unwieldy regex tables with a single declarative block served at /robots.txt. The maintenance overhead shifts from reactive bot filtering to declarative policy management.
Auditing and Verification
Adopting the standard requires rigorous verification:
- Access Log Verification: Monitor server access logs for requests originating from known IP subnets belonging to participating providers. While the initial fetch of
robots.txtis required to parse the directive, subsequent asset requests matching disallowed paths must drop to zero. - Attribution Auditing: If an enterprise's copyrighted code documentation appears in an AI model's training disclosures or direct memorization evaluations after an opt-out was established, the timestamped
robots.txtarchive serves as verifiable technical evidence of non-compliance.
For engineering organizations navigating high-scale data pipelines, model ingestion boundaries, or modernizing enterprise infrastructure, Wise Hustlers custom AI solutions provides specialized architecture design and implementation services.
Frequently Asked Questions
Does blocking __usage_ai-training remove my website from Google or Bing search results?
No. The __usage_ai-training token applies strictly to generative AI foundation model training. Traditional web search indexing crawlers, such as Googlebot and Bingbot, follow their respective user-agent blocks and standard indexing directives. Disallowing training does not affect organic search ranking or indexing.
How does __usage_ai-training interact with specific crawler directives like GPTBot?
Under Section 4.5 of the draft specification, specific directives override general directives. If your robots.txt defines specific instructions for GPTBot or ClaudeBot, the crawler follows the specific rule over the generic __usage_ai-training block. If no crawler-specific rule exists, the crawler defaults to the __usage_ai-training policy.
Is the __usage_ai-training directive legally binding or purely technical?
The directive functions as an automated, machine-readable technical expression of copyright and usage preference under RFC 9309. While its implementation among Anthropic, OpenAI, Google, and Microsoft is an industry commitment, the standard provides an unambiguous, auditable record of publisher intent that aligns with international copyright frameworks requiring machine-readable opt-outs.