OpenAI says it will honor a shared robots.txt signal allowing website operators to express generative-AI training preferences to multiple participating providers. An IETF-submitted draft uses a purpose-specific User-Agent token with existing Allow and Disallow path rules rather than new syntax. OpenAI plans implementation over the coming months while retaining current controls for individual crawlers. This is not a finished standard or a technical barrier blocking all AI collection. Operators should retain existing rules and follow each participating provider’s implementation details.
Back to Tech
OpenAI commits to a shared robots.txt signal for AI-training preferences
Voluntary adoption of a draft mechanism will not replace existing crawler controls and is not yet fully implemented.