Cloudflare launches a marketplace that lets websites charge AI bots for scraping

Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced Tuesday the launch of a new marketplace that reimagines the relationship between website owners and AI companies — ideally giving publishers greater control over their content.

For the last year, Cloudflare has launched tools for publishers to address the rampant rise of AI crawlers, including a one-click solution to block all AI bots, as well as a dashboard to view how AI crawlers are visiting their site. In a 2024 interview, Cloudflare CEO Matthew Prince told TechCrunch these products were laying a foundation for a new type of marketplace in which publishers could distribute their content to AI companies and be compensated for it.

Now, Cloudflare is bringing that marketplace to life.

It’s called Pay per Crawl, and Cloudflare is launching the “experiment” in private beta on Tuesday. Website owners in the experiment can choose to let AI crawlers, on an individual basis, scrape their site at a set rate — a micropayment for every single “crawl.” Alternatively, website owners can choose to let AI crawlers scrape their site for free, or block them altogether. Cloudflare claims its tools will let website owners see whether crawlers are scraping their site for AI training data, to appear in AI search responses, or for other purposes.

At scale, Cloudflare’s marketplace is a big idea that could offer publishers a potential business model for the AI era — and it also places Cloudflare at the center of it all. The launch of the marketplace comes at a time when news publishers are facing existential questions about how to reach readers, as Google Search traffic fades away and AI chatbots rise in popularity.

There’s not a clear answer for how news publishers will survive in the AI era. Some, such as the New York Times, have filed lawsuits against tech companies for training their AI models on news articles without permission. Meanwhile, other publishers have struck multi-year deals to license their content for AI model training and to have their content appear in AI chatbot responses.

Even so, only large publishers have struck AI licensing deals, and it’s still unclear whether they provide meaningful sources of revenue. Cloudflare aims to create a more durable system where publishers can set prices on their own terms.

The company also announced Tuesday that new websites set up with Cloudflare will now, by default, block all AI crawlers. Site owners will have to grant certain AI crawlers permission to access their site — a change Cloudflare says will give every new domain “the default of control.”

Several large publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have signed on with Cloudflare to block AI crawlers by default in support of the company’s broader goal of a “permission-based approach to crawling.”

The business model that many of these publishers relied on for decades is slowly becoming unreliable. Historically, online publishers have allowed Google to scrape their sites in return for referrals in Google Search, which translated to traffic to their sites, and ultimately, ad revenue.

However, new data from Cloudflare suggests that publishers may be getting a worse deal in the AI era than in the Google Search era. While some websites cite ChatGPT as a major traffic source, that doesn’t appear to be the case broadly.

This June, Cloudflare says it found that Google’s crawler scraped its websites 14 times for every referral it gave them. Meanwhile, OpenAI’s crawler scraped websites 17,000 times for every one referral, while Anthropic scraped websites 73,000 times for every referral.

Meanwhile, OpenAI and Google are building AI agents that are designed to visit websites on behalf of users, collect information, and deliver it back to users directly. A future in which these tools are mainstream has huge implications for publishers that rely on readers visiting their sites.

Cloudflare notes that the “true potential” of Pay per Crawl may emerge in an “agentic” future.

“What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content,” Cloudflare said in a blog post.

To participate in Cloudflare’s experimental marketplace, AI companies and publishers must both be set up with Cloudflare accounts. In their accounts, both parties can set rates at which they’d like to buy and sell a “crawl” of the publisher’s content. Cloudflare acts as the intermediary in these transactions, charging the AI company and distributing the earnings to the publisher.

Cloudflare spokesperson Ripley Park tells TechCrunch there are no stablecoins or cryptocurrency involved in Pay per Crawl at this time, even though many have suggested digital currency would be perfect for something like this.

Cloudflare’s marketplace feels like a bold vision for the future that requires a lot of publishers and AI companies to get on board. Still, there’s no guarantee publishers will get a good deal, and convincing AI firms to participate could be tough, given they’re currently scraping content for free.

Nevertheless, Cloudflare seems like one of the few companies in a position to make a marketplace like this happen.