AI Overviews Took the Click, and robots.txt Decides the Rest
Research· 6 min read· By EnterraHost Network Operations Team
Published 31 October 2026
Search results stopped being a list of places to go. On a growing share of queries, AI Overviews now occupy the top of the page with an answer written by Google, and the links below are where a smaller number of people go to check the working.
The effect on clicks is measurable. Ahrefs looked at 300,000 keywords and found that when an AI Overview appears, clicks on the top-ranking result fall by 58%. Second place loses 50.8% and third loses 46.4%. Pew Research found that a visit which included an AI summary led to a link click 8% of the time, against 15% when no summary appeared.
That is the first problem, and it is not one a site can fix. Google decides what appears at the top of its own results page.
The second problem is the one nobody checks, and it is entirely a site's own doing. Before an AI can summarize a page, a crawler has to be allowed to read it. If your robots.txt refuses that crawler, you are not in the answer, and no amount of good content changes it.
Check the second problem in about ten seconds
Our robots.txt and AI visibility checker reads a site's robots.txt the way a crawler does and reports a per-agent verdict. No signup, no email, no limit that matters.
Most tools return one number, or a list of user agents with a green tick next to each. That is not enough, because the answer depends on which job the crawler does. A search crawler that indexes pages is a different permission from a training crawler that reads them once to build a model, and blocking one while allowing the other is the whole decision.
What 500 domains say about AI Overviews and permission
We crawled the robots.txt of the 500 largest domains in the Tranco list on 30 October 2026. Of those, 264 answered with a robots.txt we could evaluate, 43 had no robots.txt at all, 40 refused the request, 29 were unreachable and 124 had no DNS answer at all. The raw rows ship with the full write-up, so the arithmetic can be checked.
Of the 264 we could evaluate, these are the counts.
| Crawler | What it does | Blocked |
|---|---|---|
| GPTBot | OpenAI, training and search | 22% |
| ClaudeBot | Anthropic | 22% |
| PerplexityBot | Perplexity | 18% |
| Google-Extended | Google, AI training and grounding | 19% |
| CCBot | Common Crawl, the public corpus most models train on | 24% |
| Bytespider | ByteDance | 25% |
| Applebot-Extended | Apple, AI training | 19% |
Eleven percent block all seven. Two percent publish a Content-Signal line, the declaration Cloudflare proposed in 2025, and 52 sites publish an llms.txt file.
The pattern is what to look at, not the individual numbers. Roughly one site in five blocks the crawler that feeds the answer it now competes with, and most of those decisions were made years ago by an agency that added a block for a crawler nobody remembered.
Why sites block without meaning to
The first is a block copied from a template. A robots.txt that says Disallow: / for a long list of bots looks careful. It usually arrived with a theme, a starter config or a migration, and nobody has read it since.
The second is a training decision applied to search. Blocking a training crawler is a defensible choice. Blocking the crawler that also grounds search results removes the site from the answer entirely, and the two are easy to confuse because the names look similar and the whole rule is one line.
The third is a CMS that writes the file. Several popular plugins rewrite robots.txt on save and drop custom rules when a setting changes. The file that was correct in March is not necessarily the file being served in November.
What to do about it
- Identify the crawler by its job rather than its name. A search crawler and a training crawler are different permissions. Decide each one deliberately and write the reason somewhere a colleague can find it.
- Allow the search side and decide the training side separately. Being in the answer requires the search crawler. Training is a commercial choice, and refusing it does not remove you from results.
- Check what is actually served, not what you saved. Fetch the live robots.txt and read the matching rule for each agent. A rule that never matches is not a rule, and a group naming a crawler with the wrong case does nothing at all.
- Re-check after every plugin update. The file is generated on some stacks, and a save can replace it.
What this does not fix
Being crawlable is necessary, not sufficient. A page that is allowed to be read still has to be worth citing, which means specific, sourced and structured so a machine can pull one sentence from it without distorting the rest. That is a writing problem, and it is slower than a robots.txt edit.
It is also worth saying what we cannot measure. We can tell you whether a crawler is allowed. We cannot tell you whether Google will choose your page for the answer, and anyone selling a guaranteed position in an AI Overview is selling something that does not exist.
The click is smaller than it was. The permission is still yours to give or withhold, and it takes ten seconds to find out which one you did.
Run the check at enterrahost.com/tools/robots-txt-tester. If you are blocking by accident, you will know before the next crawl.
Sources
- Ahrefs, AI Overviews reduce clicks by 58%, data from December 2025 across 300,000 keywords
- Pew Research Center, Google users are less likely to click on links when an AI summary appears, July 2025
- EnterraHost, AI crawler access on the 500 biggest domains, 30 October 2026
