AI readability
Guides
Newer than the rest and the least settled. The question is not whether AI is allowed in general but which agent, doing which job, on which page.
Each page answers the same three questions: why the check exists, what it costs the score, and the change that clears it. The thresholds come from the scoring method, so a number here and a number in a report cannot disagree.
Assistants now fetch pages to answer questions rather than scraping a corpus first and answering later. That moves robots.txt from a search setting to a decision about whether a machine may quote you.
What sites actually do is narrower than the argument suggests. Across 314 tests that carried an AI reading, 72% publish a Content-Signal, 78% publish an llms.txt, and 99% allow GPTBot. Blocking is the minority position, and the three guides below are about making the choice on purpose rather than by leaving a default in place.
The cost of a block is not traffic in the usual sense. A disallowed agent cannot quote you, so it also cannot cite you, and the answer it gives about your category will be assembled from the sites that did not block it. Whether that trade is worth it depends on what you sell. A publisher with a paywall and a local service business with four pages are not answering the same question.
Start with the robots.txt tester if you only want one answer today. It reads your rules the way thirteen named agents read them, and names the ones your file currently turns away.
Content-Signal, and What search, ai-input and ai-train Declare
A Content-Signal line is a sentence in robots.txt about how your content may be used, written in three parts.
Should You Block AI Crawlers? Three Choices and What Each One Costs
There are three defensible positions, and the arguments between them are usually about two different questions being answered at once.
When You Need Agent Protocols, and When a Content Site Does Not
Everything in this family answers one question: can an assistant do something on your systems, rather than read something you published?
The other groups are on the guides index, and the searchable reference for every page is the knowledge base.