What Crawlers Do Without a robots.txt, and Why It Matters

A missing robots.txt is not the emergency the card’s name suggests. A crawler that asks for the file and gets a 404 reads that as permission to crawl everything, which is what most sites want anyway.

What you lose is the conversation. This is the one file a crawler fetches before it does anything else, and it is where you say which parts of the site are not worth the crawl and where the sitemap lives.

Twenty-nine Enterramon reports across twenty domains carry this finding. It sits at low severity and the fix takes a few minutes.

A file that is allowed to be absent

The protocol is built around the idea that the file may not exist. Absence is a valid state rather than an error, and the crawler carries on without it. That is why the finding is low severity, and why a site can rank for years without ever publishing one. Two things go missing when it is absent. The first is crawl guidance, which matters on a large site and barely matters on a small one. The second is the pointer to your sitemap, and that one matters everywhere.

RFC 9309, the specification that formalised the protocol in 2022, spells out both halves of that. A 4xx response means the file is unavailable, and the crawler may then access anything on the server. A 5xx response means the opposite. The file is treated as undefined and a compliant crawler must assume the whole site is disallowed until it can read it again. A broken server blocks a site that a missing file would not. Crawlers are also told not to trust a cached copy for more than a day, which sets the ceiling on how long a stale rule can linger after you remove it.

What the robots.txt check looks for

The check requests the file at the domain root and looks at what comes back. A 200 with parseable rules passes. A 404, or a response that is not the file, raises the card.

The card reads No robots.txt, with the note that it is where you guide crawlers and point them to your sitemap. That second half is the part people skip, and it is the half worth acting on.

Where this card sits in the findings list, and what the neighbouring cards are measuring, is covered in the walkthrough of an Enterramon report.

The one line that deindexes a site

A single Disallow: / tells every crawler to stay out of the whole site. It is one line, it is valid syntax, and it is the fastest way to remove a site from search results by accident.

One line, and the whole site is out.

The other common source is a checkbox. The Reading settings screen carries one called Discourage search engines from indexing this site, and it is easy to tick while a site is being built and then forget entirely. On current WordPress it does not touch this file at all. It adds a site-wide noindex tag to every page instead, which is how a site can disappear from search results while its robots.txt looks perfectly normal.

The smaller version of the same mistake is a rule that reaches further than intended. Disallow: /news also blocks /newsletter, because the path match is a prefix rather than a whole segment.

Staging causes most of these. A development copy is often launched with the whole site disallowed so it cannot be indexed, and that line travels to production with everything else.

Writing it in WordPress

WordPress generates this file for you. Core serves a virtual one at the domain root carrying a user agent line, a rule keeping crawlers out of the admin area, an exception for the admin ajax endpoint, and since version 5.5 a Sitemap line pointing at the built-in sitemap index. It is only sent while the site is public.

The trap is that a real file on disk wins. Drop a robots.txt into your web root and the server hands that out instead, and the generated version never runs. That is how a site ends up serving a stale file somebody uploaded during a migration, while the dashboard gives no sign that anything changed.

Taking control of the file is fine, but write the whole thing, including the Sitemap line. A short file that only disallows the admin area quietly drops the pointer WordPress was adding for you.

Developers have a third route that keeps the generated file and extends it. WordPress runs its output through a filter called robots_txt before sending it, and core’s own sitemap module uses that exact filter to append the Sitemap line. Adding rules there means you keep everything WordPress generates and add to it, with nothing on disk to go stale or get lost in a migration. That is the route to hand a developer when you want a rule the dashboard does not offer.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /page/
Disallow: /cgi-bin/
Disallow: /cdn-cgi/
Allow: /wp-content/uploads/

Sitemap: https://enterramon.com/br-sitemap.xml

That is a trimmed copy of the file enterramon.com serves. Take the shape rather than the contents, because the paths that matter differ from one site to the next. If you leave the file to WordPress, the Sitemap line points at wp-sitemap.xml, which is the core sitemap on any site running 5.5 or later. Only put a different URL there if something else is generating your sitemap.

On a site behind a managed host or a CDN, the same file is often editable from that dashboard instead, and whatever sits on the origin gets ignored. Check there before editing anything on the server.

Confirming it resolves

Fetch the file in a browser. You should get plain text. If you get your homepage or an HTML error page, something is rewriting the request and the file is not being served at all.

Reading the rules back is the other half, particularly on a site somebody else set up. The Robots.txt Tester parses what your file actually says and shows which of your own pages the rules would block, which is faster than reasoning about prefix matching in your head.

Then run the test again. The card clears as soon as the path returns a parseable file, and there is nothing further to do.

The file is optional, the sitemap pointer inside it is not, and the shortest version of both is four lines.