What Cloudflare’s AI scraping move means for brands

Cloudflare, which hosts around 20% of the internet, has introduced a new system allowing websites, including publishers such as Sky News, the Associated Press, and Buzzfeed, to block AI bots from scraping their content, or charge them for the privilege. When ChatGPT launched in late 2022, it did so off the back of the internet's content, silently scraping it without consent.

Until now, AI companies have scraped everything from award-winning journalism to blogs to books and press releases without permission or payment. That era may be coming to an end with this latest update.

The power to block AI bots by default could fundamentally change the way that AI interacts with online content. For the first time, publishers will be able to opt out, or even better, opt in and charge. The introduction of a “pay per crawl” model opens the door for AI companies to finally pay for the content they rely on to function.

The truth is that this function should have been available from the start. AI systems were built on content generated by others, content that took time, integrity, and investment to produce. The “contract” that once existed between publishers and platforms like Google – visibility in exchange for traffic – has been completely bypassed by AI. Now the tables may turn.

What impact will “pay per crawl” have on internet traffic?

We’re likely to see a divide. Big publishers like Sky News and the AP will likely charge AI companies for access. They can afford to after all. Their journalism, credibility and existing reach means that people will continue to seek them out directly. Shares via TikTok and YouTube will also prevail, whatever happens to AI search traffic.

But for smaller publishers, the equation is different. They may see value in allowing AI crawlers to access their content for free in the hope that it increases exposure in results. This may even work in their favour in the short term. AI summaries could reference their pages and even drive traffic instead of referencing larger publishers as they once would have done. But here lies a deeper question: what happens to the integrity of the information if only certain content is included in AI outputs?

The journalism divide

If smaller publishers get more visibility through AI but lack the resources for deep, impartial reporting, we risk adding weight to information that is potentially less nuanced, impartial, and more fragmented. National outlets invest heavily in investigative journalism, fact-checking and comparing information for accuracy. That kind of journalism is expensive but so needed.

By comparison, AI tools value convenience, delivering concise information quickly. This raises the question: will audiences care much whether their answer came from a peer-reviewed article or a clickbait blog, as long as it sounds convincing? This could lead us to a quality of information crisis.

This new opt-in system gives the power back to publishers. But what would be the result of a retaliation from the AI companies if they refuse to pay for content, especially if other infrastructure providers don’t follow suit? And will we see a difference in the way that different countries approach this issue?

Brands may need to re-think how they view visibility

Website traffic may become a vanity metric, rather than a success metric. If AI bots are blocked, impressions may fall, but the engagement from audiences that stick around may become more meaningful. Traditional media relations tactics will prevail as the value of a well-placed brand mention in a credible feature goes up.

What happens next?

If more infrastructure companies follow Cloudflare’s lead, we could see a major realignment in how AI is fed and monetised. There may be a retaliation, but whatever happens, AI companies must now play by new rules.

There is an opportunity for PR professionals to reassert the value of high-quality journalism. AI chatbots and in search engines, and content visibility across platforms will continue to evolve, but audiences will continue to seek solutions to their problems. If the goal is to inform and empower them, then transparency, control, and fairness must lead

‍