Cloudflare has unveiled a new AI scraping policy that will enable website owners themselves to ‘decide which crawlers to allow.’ Photo Credit: Omar Lopez-Rincon
Bad news for artificial intelligence giants: Cloudflare has officially updated its AI scraping defaults in an effort to halt the “use of original content without permission.”
The self-described “leading connectivity cloud company” revealed the move – and the resulting “new business model” that it’s anticipating – in a formal release. With the change in place, “website owners can choose if they want AI crawlers to access their content,” Cloudflare summed up.
“AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow,” Cloudflare continued.
When it comes to unauthorized (and non-compensated) gen AI training, it’s hardly a secret that authors, music rightsholders, platforms like Reddit, and plenty of others are far from thrilled with the apparently widespread practice.
But Cloudflare also emphasized the traffic-related fallout associated with AI-powered search results. “AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content,” the entity indicated.
Stated differently, the AI scraping change isn’t directly tied to the music world; Cloudflare’s formal release explores expected benefits for newspaper and magazine publishers in particular. (There’s some ownership overlap here: Advance Publications owns Condé Nast as well as 30% of the aforementioned Reddit, and the CEOs of the latter two companies provided statem