Robots.txt gains a common signal for AI training preferences
OpenAI, Google, and Microsoft announce support for a shared signal in robots.txt to express website AI training preferences. The mechanism remains voluntary and complements existing controls, with implementation spread between 2026 and 2027.
OpenAI, Google, and Microsoft announced on October 30, 2026, their support for a common signal in robots.txt to allow websites to express a unique preference regarding the use of their content for AI training. The idea is simple: avoid managing a different rule for each robot while maintaining existing controls. This signal remains voluntary and is part of the IETF's work on AI-related preferences.
A single signal for multiple providers, without replacing existing rules
The core of the project lies in a shared token, __usage_ai-training, which can be included in a robots.txt file to indicate a preference on training. OpenAI explains it intends to honor it in the coming months, while Google aims for support in 2027. Microsoft, for its part, announces an extension to its own controls for training and “grounding,” meaning the support of a model on web sources to produce a response. The friction point has long been known: without a common mechanism, site publishers had to handle each crawler separately. Not exactly a walk in the park.
However, the three companies emphasize one point: this signal does not remove the proprietary controls already available. Google will keep Google-Extended, OpenAI will maintain its rules for GPTBot, OAI-SearchBot, OAI-AdsBot, and ChatGPT-User, and Microsoft plans to evolve its NoArchive and NoCache tags into new preferences. In other words, the shared standard adds to the toolbox; it does not empty it.
What the IETF draft says
The mechanism is described in an Internet-Draft titled Machine-Readable Training Preference Signal, signed by Kevin Kelley of Anthropic and dated October 30, 2026. The document does not yet have normative value: the IETF tracking tool reminds that an individual draft is not approved by the organization and has no formal status in the standardization process. Its logic remains cautious. The text adds neither new directive nor new grammar to robots.txt; it reuses the existing structure, with the already known priority rules.
Concretely, the draft associates usage preferences with specific paths. Its example shows a general ban on training on a site, except for the /press-kit/ directory. Generic groups like User-Agent: * should not be read as targeting the AI-dedicated token, and the absence of the token in a file means no preference has been expressed. The text also specifies that usage preferences and crawl signals are two distinct things: one can block training without blocking the robot’s access, depending on the case.
Why site publishers see a practical gain
For publishers, the benefit is obvious: a single rule can now be shared among multiple participating providers, instead of maintaining exceptions per actor. Google highlights that this approach responds to requests from many publishers, as well as regulators and civil society organizations, who favor more readable controls. The company adds that this path aligns with the European AI Act’s code of practice, which recognizes robots.txt as a preferred means to express machine-readable preferences.
OpenAI shares the same view, presenting the approach as a way to simplify communication between sites and AI providers. The company states that its signal does not replace already published rules and that it will continue to adapt its practices as standardization discussions progress. Google specifies that sites allowing Google-Extended remain eligible for certain Gemini experiments and clicks associated with grounding in Gemini Apps. Microsoft, finally, warns that refusing grounding may reduce brand visibility in its AI experiences, without affecting eligibility for classic Bing results or creating any license or payment agreement.
Different timelines, but a common direction
The timelines are not synchronized. OpenAI says it wants to implement the signal “in the coming months,” without a firm date. Google targets 2027. Microsoft announces, for its part, that its new preferences will take effect from February 1, 2027, with a transition period for old tags starting December 15, 2026, in Bing Webmaster Tools and a full switch after March 1, 2027, for the sites concerned. This staggered timing suggests the standard will first take the form of a gradual alignment rather than a sharp cut.
One very simple question remains: who will follow? The signal only has practical value if other actors recognize it. The three announcements of October 30, 2026, carry weight, but the mechanism still relies on voluntary adoption and the market’s ability to agree on a common interpretation. For now, consensus is moving forward. Slowly, but it is moving forward.
Key points
- OpenAI, Google, and Microsoft support a common signal on October 30, 2026.
- The token __usage_ai-training is inserted in robots.txt without new grammar.
- Google targets 2027, OpenAI a few months, Microsoft February 1, 2027.
- Microsoft replaces NoArchive and NoCache for training and grounding.
- The IETF describes an individual draft, without formal status at this stage.
- The system remains voluntary and depends on provider adoption.
By the numbers
- October 30, 2026 — date of the coordinated announcements by OpenAI, Google, and Microsoft.
- February 1, 2027 — planned start of Microsoft’s new preferences.
- December 15, 2026 — start of transition in Bing Webmaster Tools.
- March 1, 2027 — deadline to publish new preferences on Microsoft’s side.
- May 3, 2027 — expiration of the cited IETF draft in the announcement.