Cloudflare Blocks AI Crawlers, Adds Content Controls

Cloudflare, which manages about 20% of global web traffic, has begun blocking AI crawlers from accessing websites without explicit permission by default. This policy allows website owners to determine which AI bots can access their content and specify permitted uses such as training, inference, or search. Through the 'Pay Per Crawl' marketplace, currently in private beta with select publishers including The Associated Press, Condé Nast, Reddit, and Stack Overflow, site owners can set prices for AI crawlers seeking to use their data. The initiative addresses publisher concerns over uncompensated use of content by AI firms and the impact on web traffic and ad revenue. By offering granular access controls and compensation options, Cloudflare aims to create a sustainable economic model for online content in the AI era. The move could significantly influence how content creators and AI companies interact going forward.