Publishing Pulse: Google Emails Leak, Apple’s No-Search Stance, Major Sites Block GPTBot

Welcome to Publishing Pulse, your weekly source for industry updates in online publishing. Stay informed about the latest trends and breakthroughs in the ad ecosystem, content creation, SEO, AI technology, and monetization. 

If you prefer to listen to industry news, tune in to The Publisher Lab podcast. New episodes are released weekly on Thursday.

Google Exec Emails Reveal Plans to Increase Ad Revenue  

Newly disclosed emails in the U.S. antitrust case against Google provide insights into internal discussions among executives from Google’s Search, Chrome, and Ads divisions about ramping up ad revenues. 

The emails, dating from 2017 and 2019, indicate the team’s considerations for boosting ad revenues, such as increasing search queries on Chrome, and hint at potential implications on Google’s stock and personal gains for the executives. The involvement of Ben Gomes, a key figure in Google’s Search algorithm development, in these talks stands out. 

“One of the things that sticks out to me is that they [Google] are essentially managing and manipulating supply and demand,” says host Tyler Bishop on this week’s podcast episode, “and that’s one of the things that antitrust is supposed to guard against.” 

These revelations may challenge Google’s longstanding claim of keeping its Search and Advertising units distinct, emphasizing neutrality and quality in search results. This development might have a significant bearing on the antitrust lawsuit Google faces.

Apple Says They Won’t Rival Google with Their Own Search Engine 

Apple’s Eddy Cue, SVP of Services, has clarified that the tech giant doesn’t intend to launch its search engine to challenge Google. At an upcoming antitrust trial, Cue plans to endorse the Apple-Google partnership, attributing it to Google’s search engine proficiency. This collaboration, under scrutiny by the U.S. Justice Department, involves Google paying Apple roughly $10 billion annually to remain the primary search engine on Apple devices. Google’s payments, which could reach an estimated $19 billion this fiscal year, significantly benefit Apple’s services division. Cue will point out that Apple also has financial agreements with other search engines, such as Yahoo, Bing, and DuckDuckGo, emphasizing that users can effortlessly switch their default search engine. As the trial unfolds, there are contrasting views: while Google attributes its search engine dominance to its excellence, others question the ease of modifying default settings. Despite rumors, Apple maintains its relationship with Google, leveraging their partnership in negotiations.

ChatGPT Reenables Bing Search Features 

OpenAI has reintroduced the Bing search capability in ChatGPT, enhancing its ability to fetch real-time and relevant data beyond its last update in September 2021. Initially launched in May, this feature was designed to offer ChatGPT Plus subscribers more timely insights. 

Although it was briefly deactivated, it made a comeback on September 27, featuring enhancements influenced by user feedback. This includes compliance with robots.txt and improved user agent recognition for optimal website interactions. While the browsing feature is presently accessible to ChatGPT Plus and Enterprise subscribers, OpenAI intends to roll it out to all users in the near future. 

Some Major Websites Are Blocking GPTBot, Others Question Following

A significant increase has been observed in the number of major websites blocking GPTBot, OpenAI’s web crawler. Reported by Search Engine Land, 26 of the top 100 and 242 of the top 1,000 websites block GPTBot, signifying a 250% rise from last month’s 69 blocks among the top 1,000. 

New additions to the block list over the past month include popular sites like Pinterest, The Guardian, USA Today, WebMD, The Washington Post, and CBS News, among others. Interestingly, while Foursquare had previously blocked GPTBot, it has now lifted that restriction. The study, however, faced some constraints, as 67 out of the 1,000 robots.txt files weren’t analyzed. The growing trend of blocking GPTBot might stem from concerns over uncompensated data scraping or the lack of proper content attribution.

“I’d say don’t waste your time,” Tyler says to publishers who are reading this news, wondering if they, too, need to block AI crawlers. “Just because you block a crawler from your site, you essentially would have to stop visitors from logging in or authorizing their identity somehow to actually truly stop a crawler from getting your content because a crawler and a bot can perform identical behaviors as a user.”

Meanwhile, major publishers like Bloomberg and The New York Times block OpenAI’s web crawler to shield their content from AI model training. Yet, the efficacy of such measures is debatable, with some in the publishing industry viewing it more as a symbolic move than an effective strategy. 

Given that publishers distribute their content across various platforms, completely preventing access to crawlers can seem challenging. Both Google and Microsoft use web crawlers to gather data for their AI models, prompting concerns over content protection and IP rights. 

To address this, Microsoft is considering letting publishers tag their content to control its use by AI. Some view the blocking of crawlers as a negotiation strategy, aiming for compensation or licensing agreements with tech giants. However, smaller publishers might lack the clout for such negotiations and might be wary of the implications of blocking these crawlers on their own use of AI tools.

Google Allows Opt-Out For AI Training Data 

Google has rolled out a tool named Google-Extended, allowing publishers to prevent their content from training AI models such as Bard and Vertex AI generative APIs. However, the tool has ignited debates, as it auto-enrolls publishers unless they opt out, leading some to label the move as dubious. 

Accessible via the robots.txt file, the tool poses a quandary for publishers: blocking Google’s web crawlers might diminish their online visibility, subsequently affecting traffic and earnings. In response, some publishers are either suing or revising their Terms of Service to safeguard against unauthorized use of their content for AI model training. The growing unease about AI systems exploiting publisher content without permission has prompted numerous sites to block data-scraping web crawlers.

Digital Ad Sales Surge, But Traditional Media Struggles Persist

According to Adweek, ad spending in the U.S. has seen a rise, with a 4.4% growth in the second quarter year-over-year and a 2.9% uptick for the first half of the year. Consequently, the ad revenue projection for 2023 has been adjusted to an anticipated growth of 5.2%, culminating in $337 billion.

The second quarter witnessed digital media formats like search, social, and short-form video growing by 8.7%. Conversely, traditional media, such as TV and radio, dipped by 4.1%. Though there were segments like podcasting and digital out-of-home (DOOH) that grew, they couldn’t counterbalance the enduring decline in traditional media. By 2023, it’s anticipated that digital media owners will account for 69% of total ad spend, while traditional media’s predicted decline has been adjusted from 3.2% to 3.6%.

The future seems to favor digital media’s growth, with traditional counterparts likely plateauing in 2024. External factors, like a potential writers’ strike, might further drive the momentum towards digital in early 2024. Linear TV grapples with reduced pricing and viewership, leading to a predicted decrease in national TV ad revenues by 0.7% in 2024. However, local ad revenues show promise, with an anticipated 28% growth.