Major websites and AI crawlers

Key takeaways

• Blocking training does not mean blocking citation. 32.6% of the sites block at least one training crawler, but only 25% block a search crawler. Many major sites therefore distinguish between “being used to train an AI” and “being cited by an AI”.
• Structured data remains underused: nearly one site in 3 has none on its home page, although it helps search engines and AIs identify the company.
• llms.txt remains rare (14.1%): sites that adopt it early stand out at little cost.
• The average visibility score of the panel is 82/100, measured with the 33 TraceFlow Lens checks.

Method

On 18 October 2026, our crawler TraceFlowLensBot audited the home page of 147 sites chosen by the TraceFlow Labs team for their reputation, in 11 categories. For each site: reading of robots.txt (14 AI crawlers checked according to the RFC 9309 standard), of the home page (JSON-LD structured data, content readable without JavaScript), of the sitemap and of the llms.txt file, then calculation of the visibility score with the 33 TraceFlow Lens checks.

55 sites were left out because they refused our crawler or did not respond in time: liberation.fr (code 403), leparisien.fr (code 403), ouest-france.fr (code 403), lesechos.fr (code 403), lepoint.fr (code 403), lequipe.fr (code 403), jeuxvideo.com (code 403), fnac.com (code 403), darty.com (code 403), leboncoin.fr (code 403), boulanger.com (code 400), decathlon.fr (code 403), leroymerlin.fr (code 403), laredoute.fr (code 403), castorama.fr (no usable response), carrefour.fr (code 403), intermarche.com (no usable response), sephora.fr (code 403), manomano.fr (code 403), conforama.fr (code 403), amazon.fr (code 202), but.fr (code 403), bnpparibas.com (code 403), macif.fr (code 403), allianz.fr (code 403), societegenerale.com (no usable response), ameli.fr (code 403), sncf-connect.com (code 403), booking.com (code 202), gouvernement.fr (code 403), airfrance.fr (no usable response), blablacar.fr (code 403), tripadvisor.fr (code 403), accor.com (no usable response), ratp.fr (code 403), adidas.com (code 403), zara.com (code 403), hm.com (code 403), peugeot.fr (code 403), netflix.com (no usable response), reddit.com (no usable response), github.com (no usable response), stackoverflow.com (code 403), medium.com (code 403), notion.so (no usable response), canva.com (code 403), nytimes.com (code 403), reuters.com (code 401), koaci.com (code 500), apanews.net (code 403), bceao.int (no usable response), jumia.ma (no usable response), afdb.org (code 403), lematin.ma (code 403), ecobank.com (no usable response).

This panel is not a representative sample of the whole web: it describes the practices of well-known sites at a given date. Its 92 results also feed the live Barometer, together with the audits run by our users.

Reusing this data

Free, under the Creative Commons BY 4.0 licence, by citing “TraceFlow Lens study, October 2026” with a link to this page.

Frequently asked questions

What share of major websites block ChatGPT?

In our panel of 92 sites, 23.9% block GPTBot, the crawler that collects texts to train OpenAI models, and 12% block OAI-SearchBot, the crawler that lets ChatGPT Search cite pages.

Do major websites block AI search engines?

25% of the sites in the panel block at least one AI search or browsing crawler (ChatGPT, Claude, Perplexity or Amazon), which prevents them from being cited in their answers.

How many major websites have an llms.txt file?

14.1% of the sites in the panel publish an llms.txt file, a recent format that presents a site to AIs.