Cloudflare wants Google to change its AI search crawling. Google likely won’t.

After Cloudflare started testing new features that would allow websites to block AI crawlers or require payment for scraping, the tech company immediately faced questions over the logistics of the plan.

In particular, website owners and SEO experts wanted to know how Cloudflare planned to block Google’s bot from scraping sites to fuel AI overviews without risking blocking the same bot from crawling for valuable search engine placements.

Last week, a travel blogger raised questions about the blocking and so-called pay-per-crawl features and pushed Cloudflare CEO Matthew Prince to respond on X (formerly Twitter):

“We will get Google to provide ways to block Answer Box and AI Overview, without blocking classic search indexing, as well,” Prince said. Asked if that was even possible, Prince doubled down, responding, “it is. #staytuned”

In another post responding to a search engine optimization specialist, he claimed that Cloudflare was in “encouraging” talks with Google that he hopes will result in Google separating its crawlers to better work in Cloudflare’s system. But if those talks go nowhere, he revealed Cloudflare is pushing for a law to be passed that’s considered a “very viable option” in “many jurisdictions.”

“Worst case we’ll pass a law somewhere that requires them to break out their crawlers and then announce all routes to their crawlers from there,” Prince said. “And that wouldn’t be hard. But I’m hopeful it won’t need to come to that.”

Ars could not immediately find any legislation that seemed to match Prince’s description, and Cloudflare did not respond to Ars’ request to comment. Passing tech laws is notoriously hard, though, partly because technology keeps advancing as policy debates drag on, and challenges with regulating artificial intelligence are an obvious example of that pattern today.

Google declined Ars’ request to confirm whether talks were underway or if the company was open to separating its crawlers.

Although Cloudflare singled out Google, other search engines that view AI search features as part of their search products also use the same bots for training as they do for search indexing. It seems likely that Cloudflare’s proposed legislation would face resistance from tech companies in a similar position to Google, as The Wall Street Journal reported that the tech companies “have few incentives to work with intermediaries.”

Additionally, Cloudflare’s initiative faces criticism from those who “worry that academic research, security scans, and other types of benign web crawling will get elbowed out of websites as barriers are built around more sites” through Cloudflare’s blocks and paywalls, the WSJ reported. Cloudflare’s system could also threaten web projects like The Internet Archive, which notably played a crucial role in helping track data deleted from government websites after Donald Trump took office.

Among commenters discussing Cloudflare’s claims about Google on Search Engine Round Table, one user suggested Cloudflare may risk a lawsuit or other penalties from Google for poking the bear.

Ars will continue monitoring for updates on Cloudflare’s attempts to get Google on board with its plan.

留下评论

您的邮箱地址不会被公开。 必填项已用 * 标注