Should You Block AI Crawlers? Three Choices and What Each One Costs
There are three defensible positions, and the arguments between them are usually about two different questions being answered at once. One is whether an assistant may cite your page when it answers; the other is whether your page may be used to train a model. They have different costs, so a site can reasonably answer them differently.
Before the choice, one correction to a common assumption. Being read by an AI crawler does not raise or lower your search rankings. Google's search crawler and the crawler behind Google-Extended are separate controls, and blocking the AI one does not reduce your visibility in ordinary search results. What it can reduce is your presence in answers, which is a different channel with its own value.
Option one: allow everything
The default for most sites, because nothing in robots.txt says otherwise. Assistants can read pages, quote them, and train on them.
What you get: the widest possible presence in AI answers, at no effort. If a model answers a question your page covers, your page is available to be read and cited.
What you give up: control over training use, and no ability to tell a citing crawler from a training crawler after the fact. For a site whose value is being found, this is often the right trade, and it is what a blog, a documentation set or a brochure site usually wants.
Option two: citation yes, training no
The position most publishers land on once they think about it, and the one our own sites take. Search and answer-time reading are welcome; training is not.
What you get: you stay citable and findable while declining the use that is hardest to undo. A model trained on your content produces answers you cannot inspect, correct, or attribute.
What you give up: nothing measurable today, and the honest caveat is that the line is a declaration rather than an enforcement. It is honored by the crawlers that read robots.txt and ignored by the ones that do not.
User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /
The Content-Signal guide explains what each value means and why the line belongs inside one user-agent group.
Option three: block every AI crawler
The right answer for some businesses, and the wrong default for most. It is defensible when the content is the product, when licensing is a revenue line, or when a site is being crawled hard enough to cost real money.
What you get: no AI use of any kind, and the strongest signal you can send without a firewall rule.
What you give up: every citation, including the ones that would have sent a reader to your page. You also take on maintenance, because the list of crawler names changes and a name you forgot is a crawler you did not block. And it is worth knowing that blocking by user-agent is trivially spoofable; only a verified-bot control or a firewall rule actually enforces it.
# English-language assistants User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / # Chinese assistants. Names matter here: a crawler that is not # listed is not blocked, and the Chinese labs use their own. User-agent: Bytespider Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: QwenBot Disallow: / User-agent: KimiBot Disallow: /
The Chinese crawlers are the ones most often forgotten, and they are not an afterthought: ByteDance, DeepSeek, Alibaba's Qwen and Moonshot's Kimi each run their own agents, under their own names, so a list written for OpenAI and Anthropic blocks none of them. Kimi alone publishes three - KimiBot for training, Kimi-SearchBot for its search index and Kimi-User for pages a person asked it to read - in its own crawler policy. Blocking only KimiBot still lets Kimi index you.
A framework rather than an answer
Ask three questions, in this order.
Do you want to be quoted? If a reader arriving from an assistant is valuable, keep citation open. If your business is selling the content itself, close it and accept the loss of that channel.
Is training a commercial risk? If your content is a product, or your licensed material is a revenue line, say no to training even while saying yes to citation. If it is a support article that exists to be found, training is a much smaller concern.
Can you enforce what you declare? If you cannot block at the network edge, a robots.txt statement is a preference and should be written as one. If you have a CDN with bot verification, the declaration and the enforcement can match, and that is the ideal state.
Our own position, stated once
EnterraHost publishes search=yes, ai-input=yes, ai-train=no, and publishes an llms.txt so an assistant can find the pages worth reading. The guides exist to be found and quoted; training is a different use and we decline it. You may reasonably decide otherwise, and our report will not mark you down for it: it reports which agents are blocked and names the rule, because only the owner knows whether that was the intention.
