Now Reading: Google crawler gives AI Gemini unfair edge, faces UK and Cloudflare pushback
-
01
Google crawler gives AI Gemini unfair edge, faces UK and Cloudflare pushback
Google crawler gives AI Gemini unfair edge, faces UK and Cloudflare pushback
Google’s web crawler, used to index pages for search, now also feeds the company’s Gemini AI model, giving it an unfair advantage. Cloudflare CEO Matthew Prince says Google’s bot sees 3.2 times…
The Story in Brief
Google’s web crawler, used to index pages for search, now also feeds the company’s Gemini AI model, giving it an unfair advantage. Cloudflare CEO Matthew Prince says Google’s bot sees 3.2 times more of the web than OpenAI’s, and warns that bundling search and AI scraping makes it hard for publishers to block AI training without losing search traffic.
AI agents are set to generate 57% of web traffic in 2026, raising fears that human-created content will become economically unviable. The UK’s Competition and Markets Authority has ordered Google to give websites a clear choice to block AI use without search penalty. Google is testing a global opt-out setting, but its crawlers remain technically bundled for both purposes.
The Indian Opinion
Google’s claim that bundling search and AI scraping is efficient ignores the real cost: human creators subsidising a machine that replaces them. The ‘free ride’ narrative is one-sided, Google did build the web’s biggest index, but the company’s own internal slides show it chose monetisation over choice. The real test is whether Google’s promised global opt-out setting is genuine or another half-measure. Will Cloudflare’s September deadline force real decoupling, or will Google find a workaround?
Source: economictimes.indiatimes.com
This story was synthesised by AI from the source linked above.
