"In recent years, news sites have been receiving an influx of traffic from a new kind of visitor: AI crawlers. From Googlebot to OpenAI’s “GPTbot,” Anthropic’s “Claudebot” and Meta’s “ExternalAgent,” these crawlers all have one thing in common: a persistent need for data to train Large Language Models (LLMs), develop AI assistants, and feed AI-powered search engines. These new visitors are noteworthy especially when we consider that around 30% of the global web traffic today comes from bots. News sites are attractive destinations as they produce timely, reliable information that can help to improve the quality of their AI models and outputs. This surging demand for AI data presents new opportunities—but many challenges—for media organizations. It raises important legal and ethical issues around unauthorized use, copyright infringement, and privacy violations. But the very survival of journalism hangs in the balance, too: without mechanisms in place to signal and enforce permissions on AI crawlers, media organizations are unable to effectively protect and monetize the value of their work as it gets scraped, repackaged, and redistributed by AI systems. This gap risks accelerating the sustainability crisis facing media outlets that are already grappling with declining visibility and traffic from digital platforms—what many publishers have already called a “traffic apocalypse”— and the collapse of traditional business models based on advertising.
News sites are becoming increasingly aware of this growing challenge, but there are currently few ways to manage AI crawlers and none of them are completely foolproof. Some sites are implementing hard paywalls, while others are launching lengthy lawsuits claiming copyright infringement. Technical responses are also emerging: in July 2025, the web security and infrastructure provider Cloudflare announced that its protections would block AI crawlers by default and help to enforce a permission-based model. Content marketplaces—like the one piloted by Microsoft or startups such as ProRata.ai and Tollbit—and new data solutions developed by the likes of Flexolmo and OpenMined are also emerging as ways to give content producers more control over how their content gets accessed, used, and compensated by AI companies. But many more still resort to managing content scraping by AI crawlers by implementing the Robots Exclusion Protocol (known as robots.txt)." (Introduction, page 4)