Major publishers are reconsidering how much access they should give Google's AI systems as search traffic continues to fall, according to a report by The Wall Street Journal.
Several media companies are evaluating whether to restrict Google's AI crawlers from accessing their content, with some even discussing the possibility of blocking Google's primary web crawlers altogether — a move that could significantly affect how their content appears in search results.
The debate reflects mounting concerns among publishers that Google's AI-generated summaries are reducing the number of users clicking through to their websites, hurting audience growth and advertising revenue.
ALSO READ | The Death Of The Click: How India's Proposed AI Payment Protocol Will Upend Digital Commerce
Reddit Among Publishers Exploring Restrictions
According to the report, Reddit has internally discussed limiting Google's access to its content for AI-related purposes.
Such a move would mark a significant shift, given that Google agreed to pay Reddit about $60 million annually under a 2024 licensing deal that allows the tech giant to use Reddit's content to train its AI models.
The report also said several other publishers are weighing similar options.
Media Executives Voice Concerns
Executives from leading media organisations have openly expressed concerns over Google's AI strategy.
Reuters President Paul Bascobert said the company is assessing the economic trade-offs between appearing in search results and allowing its content to be used in AI-generated summaries.
Gannett CEO Mike Reed said publishers need to "take a stand" against the current model, while Neil Vogel, chief executive of Dotdash Meredith, said completely blocking Google's crawlers is "100% on the table."
According to the report, Politico and The Economist are also evaluating how they want to handle Google's web crawlers.
The Trade-Off Facing Publishers
The report noted that publishers can block Google's AI training crawler, known as Google-Extended, preventing Google from using their content to train AI models such as Gemini.
However, publishers that want their content to appear in both traditional Google Search and AI-generated search results generally need to continue allowing Google's primary web crawler to access their websites.
Industry tracking platform BotDetector said publishers that want to avoid appearing in Google's AI Overviews can also use the "nosnippet" tag, although doing so could reduce their visibility in search results.
A more aggressive option is the "noindex" tag, which prevents pages from being indexed by Google Search altogether.
According to the report, if a significant number of publishers adopted that approach, it would mark a major escalation in the standoff between publishers and Google.
ALSO READ | Google Updates Cost-Effective Gemini Models; Release Of Flagship Pro Model Delayed
Google Defends AI Search
Responding to the report, a Google spokesperson told The Wall Street Journal that the company's AI-powered Search features continue to drive traffic to websites.
The spokesperson said Google's AI delivers "billions of clicks to the web every week," adding that the company highlights links to publishers' websites and provides website owners with tools to control how their content is used.
Essential Business Intelligence, Sharp Market Insights, Practical Personal Finance Advice, Daily Fuel, Gold and Silver Prices and Latest Stories — On NDTV Profit.
