Now Reading: Google bows to pressure, lets websites block AI scraping
-
01
Google bows to pressure, lets websites block AI scraping
Google will roll out a global setting letting websites opt out of its AI answers without hurting their search rankings, a spokesperson confirmed, following pressure from the UK's Competition and Markets Authority.…
The Story in Brief
Google will roll out a global setting letting websites opt out of its AI answers without hurting their search rankings, a spokesperson confirmed, following pressure from the UK's Competition and Markets Authority. The test begins after the UK ordered Google in June to give website owners a clear choice to block content from AI products while staying in search results.
Cloudflare, which handles traffic for a fifth of the web, will block mixed-purpose crawlers by default for ad-supported customers from September 15. CEO Matthew Prince said Google's crawler sees 3.2 times more of the web than OpenAI's and 4.8 times more than Microsoft's. Cloudflare says AI agents generated over 57% of web traffic versus 42% for humans in 2026, the first time machines surpassed humans.
The Indian Opinion
Google's concession is welcome, but it should never have taken a regulator's ultimatum to do the right thing. The narrative that scraping is inevitable progress masks a simple truth: publishers fund the web, and Google profits from it. Watch whether the rollout actually protects small sites or quietly favours big media. The real test is whether the opt-out remains genuinely neutral in rankings, not just in the UK but everywhere Google operates, including India. Will Indian publishers get the same protection? That question deserves an answer.
Sources (2): economictimes.indiatimes.com, hindustantimes.com
This story was synthesised by AI from the 2 sources linked above.
Updated: this story now draws on 2 sources.
