What 500 of the Biggest Domains Do About AI Crawlers
Research· 4 min read· By EnterraHost Network Operations Team
Published 30 October 2026 ·Updated 31 October 2026
Twenty-two percent of the largest sites that publish a robots.txt file block GPTBot, one of the four main AI crawlers we asked about. Eleven percent block every AI crawler we asked about, including the one Google uses to control training. And three percent publish a Content-Signal line, the declaration Cloudflare proposed in 2025 and that almost nobody has adopted yet.
Those numbers are from one crawl, run on 30 October 2026, of the 500 largest domains in the Tranco list. The method and the exclusions are below, and the raw rows ship with the article so the arithmetic can be checked rather than believed.
What we actually did
We took the top 500 entries of the Tranco list (list PYK6J, built 29 October 2026), fetched https://<domain>/robots.txt for each, and evaluated the result per user-agent using the group matching and longest-match precedence defined in RFC 9309. A crawler is "blocked" when the rules that apply to it disallow /. The rules that apply are a group naming the agent if one exists, otherwise the * group.
Requests carried a browser user agent string with our own token appended, so a firewall saw an attributable request rather than an anonymous one. Timeouts were 12 seconds, redirects were followed at most once to http, and a site that refused or failed is recorded as such rather than retried until it agreed with us.
Seven agents were counted - GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider - which is the English-language set plus ByteDance. The robots.txt and AI visibility tester now asks fifteen, including DeepSeek, Alibaba's Qwen and Moonshot's Kimi, and this study predates that. It is stated rather than backfilled. The percentages below are over seven agents, and the Chinese labs beyond ByteDance are not in them.
The half of the sample that is not a website
The first thing the crawl found is that Tranco's top 500 is not a list of 500 websites. 124 of them have no A record at all. Among them are akamai.net, cloudfront.net, trafficmanager.net and gvt1.com. They are infrastructure and CDN domains that resolve to nothing because they were never meant to be visited. A domain with no address is not a site that blocks crawlers. It is not a site.
That leaves 376 real websites.
| State | Domains | Share of 376 |
|---|---|---|
| robots.txt served and read | 264 | 70% |
| No robots.txt (404) | 43 | 11% |
| Refused our request (403 or 429) | 40 | 11% |
| Unreachable (timeout, 5xx) | 29 | 8% |
Forty sites would not serve their own robots.txt to a request that identifies itself. That is worth noticing on its own. The file exists to be read by machines, and a firewall that refuses a crawler cannot be told apart from a firewall that refuses everyone.
Which AI crawlers are blocked, among the 264 we could read
| User-agent | Operator | Blocked | Share |
|---|---|---|---|
| GPTBot | OpenAI | 59 | 22% |
| ClaudeBot | Anthropic | 57 | 22% |
| PerplexityBot | Perplexity | 47 | 18% |
| Google-Extended | Google (training control) | 51 | 19% |
| Applebot-Extended | Apple (training control) | 51 | 19% |
| CCBot | Common Crawl | 63 | 24% |
| Bytespider | ByteDance | 67 | 25% |
Thirty percent block at least one of the seven, and 11% block all seven. The sites that block everything are the ones you would guess from the copyright argument and a few you might not. Facebook, Pinterest, TikTok, Reddit, Baidu, Weibo and Naver all block everything, and so does a run of news and reference publishers, among them Reuters, Bloomberg, CNBC, the BBC, The New York Times, Nature and IMDb. ChatGPT's own domain blocks every one of them, which is a sentence worth reading twice.
Almost nobody separates training from citation
The distinction the arguments are usually about, letting an assistant read the page and cite it while refusing training, barely appears in the data. Two sites allow GPTBot and block Google-Extended. Ten do the opposite, blocking GPTBot and allowing Google-Extended. Everyone else treats the whole category as one decision.
That is the gap this whole workstream exists for. A robots.txt can express the split perfectly well. The sites that care enough to write rules are not yet writing that one.
Content-Signal: three percent, a year in
Eight of the 264 publish a Content-Signal line. The syntax is simple and the placement is the part people get wrong. It belongs inside a user-agent group, not at the top of the file.
User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /
Eighteen percent publish an llms.txt. We counted a file only when it returned 200 with markdown content and no HTML. A catch-all page that answers 200 for every path is not a declaration, and pretending otherwise would have tripled that figure.
What this does not tell you
- It is one snapshot. robots.txt changes, and a site that blocked GPTBot in October may allow it in January. The CSV is dated for that reason.
- It is one vantage point. Every request came from a single region. A CDN that answers us differently elsewhere would not show up.
- robots.txt is a request, not a wall. A blocked crawler can ignore the file. The sites that mean it block at the network edge instead, and that is invisible from here.
- The sample is big western domains. Tranco is weighted toward sites Chrome and Cloudflare users visit. It says nothing about small sites, and nothing about your site.
- Forty refusals are not forty blocks. A 403 to our request could be a WAF rule against any automation, and we cannot tell which from outside.
Where your site sits
The same check runs on your own domain in about a second, and it scores the things this article counted: which agents may read you, whether a Content-Signal is published, whether an llms.txt exists, and whether the sitemap and Link headers help an assistant find the right page.
Run the robots.txt and AI visibility test, or read what blocking an agent actually costs, what those blocks cost when the answer sits above the link in AI Overviews and the click before deciding. The raw data for this article is a CSV: 500 rows, one per domain, with the state of each and the verdict for all seven agents.
Use it. The dataset is published under CC BY 4.0 - copy it, chart it, quote it, no permission needed. If you cite it, credit it as Enterramon, "What 500 of the Biggest Domains Do About AI Crawlers", 30 October 2026 and link this page, so a reader can check the method rather than take the number on trust.
