Meta removes disputed clause on site scraping
Meta has removed a wording from its commercial terms that seemed to allow the use of site content, including news, to train its AIs. At the same time, publishers are tightening controls against scraping bots.
Meta removed from its commercial terms a phrase that appeared to grant it the right to use content from sites, including news sites, to develop and train its AI products, according to The Verge on October 17, 2026. The company says this text was related to its advertising and business products, not an authorization to override site preferences. This clarification comes amid growing distrust of scraping bots, as publishers seek to regain control over their data.
Une formulation devenue trop ambiguë pour rester en place
The removed phrase stated that Meta "may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products, services, and features provided under these Commercial Terms and any other applicable supplemental terms." Several observers, including Neiman Lab and James Grimmelmann, digital law professor at Cornell, saw this as a potential legal basis to scrape site content and reuse it for AI, including bypassing robots.txt. Meta disputed this interpretation. Andy Stone and Daniel Roberts initially stated there was "no change to how Meta uses web crawlers; any suggestion to the contrary is false." Roberts later clarified: "We’ve heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we’ve clarified that in the terms themselves. It does not change how Meta uses web crawlers. As always, content owners can use robots.txt to signal their preferences. This is not about scraping news sites, and these updates don’t give us new rights that supersede those preferences."
In other words, the company preferred to remove the phrase rather than maintain ambiguity. A simple choice, often more effective than prolonged public relations efforts.
Les éditeurs ferment la porte aux crawlers d’IA
This retreat by Meta comes as more and more sites toughen their policies toward bots. An analysis published on Dev.to on August 2, 2026, covering the 2,000 largest web domains, shows that 292 out of 1,530 domains with accessible robots.txt files, or 19.1%, block at least one of the four major AI crawlers: GPTBot, ClaudeBot, Google-Extended, or PerplexityBot. A year earlier, such blocking was marginal.
Cloudflare has also scaled up. Since September 15, 2026, any new domain registered on its platform has bots classified as "Training" or "Agent" blocked by default on pages containing advertising, while indexing bots related to search remain allowed. The rule was then extended to existing free-tier clients and new sites of already registered clients. Cloudflare distinguishes three categories: Search for indexing, Training for training data collection, and Agent for bots acting in real time for a user.
Quand les robots prennent plus qu’ils ne renvoient
Publishers are not only protesting the principle. They also look at the numbers. According to Cloudflare Radar data cited in the sources, Anthropic’s ClaudeBot crawled 23,951 pages for every referral to a publisher in Q1 2026, then 1,917 to 1 in July 2026. OpenAI’s GPTBot showed 1,276 to 1 from January to March 2026, then about 251 to 1 in July, compared to 1,104 to 1 a year earlier. The problem is arithmetic: a bot that massively collects without sending readers back costs infrastructure without generating advertising revenue.
Sites no longer block bots one by one. The Dev.to study notes that 75% of domains blocking at least one bot block three or four at once, indicating an increasingly collective policy. And when a site wants to protect training but not indexing, the situation becomes complicated: Googlebot, Bingbot, or Applebot combine several uses, which can make blocking broader than intended.
Le web public entre indexation et prélèvement
The dividing line is now clear. Traditional search engines provided a service in exchange for their crawling. AI crawlers extract data to feed models and, sometimes, agents that consult the web live. In this context, the neutrality of robots.txt looks less and less like a tacit agreement and more like a polite request. This is significant: the battle is about publishers’ ability to decide who can read their pages and for what purpose.
For Meta, the wording correction avoids fueling suspicion of an implicit right to scrape news sites. For the rest of the sector, it mainly reminds that the robot war is now won as much in terms of use policies as in crawler code.
Key points
- Meta removed a disputed clause on October 17, 2026.
- 19.1% of studied sites block at least one AI crawler.
- Cloudflare blocks "Training" and "Agent" bots by default.
- ClaudeBot reached 23,951 pages per 1 referral in Q1 2026.
- Robots.txt remains a preference, not an absolute technical barrier.
By the numbers
- 1,530 domains — sample accessible via robots.txt, Dev.to analysis of August 2, 2026.
- 292 domains — sites blocking at least one of the four major AI crawlers.
- 19.1% — share of domains blocking at least one crawler, Dev.to study.
- 23,951 pages per 1 referral — ClaudeBot ratio in Q1 2026, Cloudflare Radar.
- 1,276 pages per 1 referral — GPTBot ratio from January to March 2026, Cloudflare Radar.