The Meta AI training clause has been removed from the commercial terms taking effect on 30 October 2026, after publishers objected. The word “train” disappeared from the preview late on Friday 16 October, but the crawling permission itself stayed. Meta’s own documentation lists five separate web crawlers.
What Meta Removed and What Stayed
Meta deleted one word and kept the permission that word described. “Train” is no longer among the listed uses; crawling websites linked to Meta’s business products still is.
The original language let Meta use automated software to access, copy and re-crawl websites associated with business accounts on Facebook, Instagram and WhatsApp, and covered using that content to train Meta’s own products. After criticism, the preview text was revised late on Friday 16 October 2026, according to The Next Web, which reported that the terms “no longer uses the word ‘train'”. The revised version still covers data gathered by web scrapers, bots, spiders, crawlers and user agents.
Meta’s position is that nothing of substance changed because nothing of substance was ever intended. The company said the crawling terms only ever applied to its advertising and business products, and that it is not changing how its web crawlers operate. Communications representatives Daniel Roberts and Andy Stone made that case publicly, with Roberts saying robots.txt files will still keep paywalled articles and private content from being crawled.
The acceptance mechanism has not changed either. Businesses agree to the terms by continuing to use Meta’s commercial products — there is no separate consent step, which is what the original terms were criticised for when they were published.
Meta’s Five Crawlers, and Which One the Terms Are About
Meta does not run one crawler; it runs five, each with its own user-agent token and its own purpose. This matters because the one most site owners have already blocked is not the one Meta says the commercial terms concern.
| User agent | What Meta says it does |
|---|---|
| facebookexternalhit/1.1 | Crawls pages shared on Meta apps to collect the title, description and thumbnail |
| meta-webindexer/1.1 | Analyses content to improve Meta AI search results and to cite and link sources in Meta AI answers |
| meta-externalads/1.1 | Crawls the web to improve advertising and other business-related products and services |
| meta-externalagent/1.1 | Crawls the web to train foundation AI models, or to improve products by indexing content directly |
| meta-externalfetcher/1.1 | Fetches individual links when a user requests them, supporting agentic AI features |
Read Meta’s defence against that list and it points at meta-externalads, the advertising and business-products crawler. The crawler whose stated job is training foundation models is meta-externalagent — a different token, which a robots.txt rule naming one does not cover.
Why Blocking the AI Crawler Is Not Enough
Two of Meta’s five crawlers can ignore robots.txt by Meta’s own account, and a rule written for one token does nothing about the other four. Site owners who added a single disallow line for the AI-training crawler have narrower protection than they think.
Meta’s developer documentation states plainly that “The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user”, and that facebookexternalhit “might bypass robots.txt when performing security or integrity checks”. No such exception is listed for meta-webindexer, meta-externalads or meta-externalagent, so standard rules apply to those three.
There is also a timing problem. Meta’s documentation asks site owners to allow up to 24 hours for robots.txt changes to take effect. With the terms starting on 30 October 2026, a file edited on the 29th may not be honoured before the terms do.
How to Block Meta’s Crawlers Before 30 October
Decide which of the five uses you object to, then write one rule per token, because Meta treats them separately. The order below moves from the AI uses to the ones that affect how your links appear on Meta’s platforms.
- Block model training: Add a group for meta-externalagent with a disallow for the paths you want excluded. This is the crawler Meta names for training foundation models.
- Block advertising and business crawling: Add a separate group for meta-externalads. This is the crawler that matches what Meta says the commercial terms cover, and the one most blocking guides omit.
- Decide about Meta AI search: meta-webindexer feeds Meta AI search results and the citations and links in Meta AI answers. Blocking it removes your site from those answers as well as from that indexing.
- Weigh link previews before touching facebookexternalhit: This is the crawler that builds the title, description and thumbnail when someone shares your page on Facebook or Instagram. Blocking it degrades those previews, and Meta says it may bypass robots.txt for integrity checks anyway.
- Accept that user-requested fetches may continue: Meta states that meta-externalfetcher may bypass robots.txt, because the fetch was asked for by a person using an agentic feature.
- Allow 24 hours: Make the change by 28 October 2026 at the latest if you want it in force when the terms begin.
Blocking at the server or firewall level, by rejecting requests whose user-agent string matches, is the only method that does not depend on a crawler choosing to honour the file. Meta has not said it blocks requests on that basis itself.
What Publishers and Lawyers Say Is Still Open
The removal of one word has not settled the question it was removed over. Three objections survive it.
- Training is not banned, only unlisted: News/Media Alliance spokesperson Sam Quigley said training was taken out of the listed uses but not expressly prohibited, and that “It’s still potentially on the table.” The Alliance called the deletion a step in the right direction while saying the terms still give Meta broad crawling permission without hard contractual guarantees.
- The terms do not promise robots.txt will be respected: Quigley made that point, and Santa Clara University law professor Eric Goldman told Nieman Lab the language “doesn’t expressly say that Meta will respect those robots.txt instructions”. Meta’s public assurances on robots.txt sit outside the contract.
- The choice itself is the objection: Cornell law professor James Grimmelmann argued the policy leaves newsrooms choosing between supplying original reporting for AI training and losing their social media distribution. Before the revision, he read the terms as granting permission to train on any content from a news publisher’s site.
Danielle Coffey, president and chief executive of the News/Media Alliance, criticised the terms publicly on 16 October 2026, the day before the wording changed. The pattern is familiar from other recent publisher-facing AI terms, including Google’s contribution pilot, where payment is optional and use is not, and it sits against a legal backdrop in which an AI training fair-use ruling was upheld on sealed reasoning.
Status as of 19 October 2026
As of 19 October 2026, Meta’s commercial terms page shows an effective date of 6 March 2024 with a banner stating that updates take effect on 30 October 2026. The live text carries no crawling or AI-training clause; the disputed language belongs to the version that starts on that date.
Meta has not published a redline of what changed, has not committed in the contract to honouring robots.txt, and has not said whether crawled business-site content may be used for training under the revised wording. No regulator has opened a public action over the terms. The 30 October start date stands.
Frequently Asked Questions
Did Meta Cancel the New Commercial Terms?
No. The terms still take effect on 30 October 2026. Meta removed the word “train” from the listed uses in the preview text; the rest of the terms, including the crawling permission, remain.
Which Meta Crawler Trains AI Models?
Meta’s documentation names meta-externalagent/1.1 as the crawler that crawls the web to train foundation AI models or to improve products by indexing content directly.
Does Meta Honour Crawler Blocking Rules?
Meta says it will for most crawlers, but the contract does not say so. Meta’s own documentation states that meta-externalfetcher may bypass robots.txt for user-requested fetches, and that facebookexternalhit might bypass it during security or integrity checks.
Do I Have to Agree to the Terms?
There is no separate consent step. Businesses accept the terms by continuing to use Meta’s commercial products after they take effect.
Can Meta Still Train AI on My Website After the Change?
That is unresolved. The News/Media Alliance says training was removed from the listed uses but not expressly banned, and Meta has not stated whether crawled business-site content may be used for training under the revised wording.
