Sites without a working robots.txt by popularity (HTTP Archive, Sept 2026)
| Chrome rank | Origins | No file (404/410) | HTML page | 401/403 | 429/5xx | No response | Not working total | Median working file (bytes) |
|---|---|---|---|---|---|---|---|---|
| Top 1K | 754 | 4.9% | 3.2% | 1.5% | 0.1% | 1.2% | 10.9% | 1,347 |
| 1K to 10K | 6,746 | 8.2% | 3.4% | 2.6% | 0.1% | 1.2% | 15.5% | n/a |
| 10K to 50K | 29,679 | 11.7% | 5.1% | 2.5% | 0.2% | 1.3% | 20.8% | n/a |
| 50K to 1M | 745,671 | 14.4% | 5.5% | 1.3% | 0.2% | 1.0% | 22.5% | 411 |
| Beyond 1M | 14,687,628 | 11.6% | 3.8% | 0.5% | 0.3% | 1.0% | 17.3% | 200 |
| All | 15,470,478 | 11.7% | 3.9% | 0.6% | 0.25% | 1.0% | 17.5% | 206 |
Source: PageSpeed Matters, Sept 2026. Medians are approximate; n/a means no median was reported for that bucket.
Summary
A robots.txt file is the first thing a crawler asks for on a site. It tells search engines which paths to skip and often where the sitemap lives. When the request fails, crawlers have to fall back on their own defaults, and the site loses that bit of control.
We checked the robots.txt response for every origin in the September 2026 HTTP Archive crawl. Of 15,470,478 origins, 17.5%, about 1 in 6, have no working robots.txt file. The failure rate is highest not on tiny sites but on mid-size ones ranked 50K to 1M by Chrome popularity.
What we measured
We looked at the robots.txt response for every origin in the September 2026 HTTP Archive crawl, all 15.5M of them, and grouped origins by their Chrome popularity rank.
A file counts as working when the server returns it as a text file. Everything else counts as not working, split into five groups: a 404 or 410 (no file), an HTML page served instead of a text file, a 401 or 403 (blocked by a login or firewall), a 429 or server error, and no response at all.
Two limits apply. HTML responses are usually catch-all pages, not deliberate blocks, and server errors can be temporary, so a single crawl may catch a site on a bad day. File size medians are approximate.
One in six origins has no working robots.txt
Across all 15,470,478 origins, 17.5% have no working robots.txt. Most of those simply have no file: 11.75% return a 404 or 410.
The rest break in other ways. 3.91% return an HTML page instead of a text file, usually a homepage or error template that the server sends for any unknown path. 0.58% block the file with a 401 or 403. 0.25% return a 429 or a server error, and 0.99% don't respond.
These failures are not equal. A missing file is the mildest case: Google's documentation says it treats most 4xx responses as if there were no robots.txt, so it crawls without restrictions. An HTML page in place of the file, or a file that sits behind a server error, is messier, because the site has lost a clear signal it could easily send.
17.5%
Share of the 15,470,478 origins in the September 2026 HTTP Archive crawl with no working robots.txt file.
Mid-size sites fare worst
The failure rate rises as sites get less popular, peaks in the middle and then eases off again. 10.9% of the top 1K origins lack a working file. That rises to 22.5% of the 745,671 origins ranked 50K to 1M, then drops back to 17.3% of the 14.69M origins ranked beyond 1M.
The worst single bucket is 100K to 500K, where 23.0% of 309,944 origins have no working robots.txt. The chart shows the curve: 14.6% for 1K to 5K, 16.2% for 5K to 10K, 20.8% for 10K to 50K, then 22.6%, 23.0% and 22.2% through the 50K to 1M range.
Our data does not show why, but a likely reason is that mid-size sites are big enough to run custom stacks and many hosts, while often lacking someone whose job is to check the basics on each one.
How sites fail depends on their size
Mid-size sites mostly fail by having no file at all. 14.4% of origins ranked 50K to 1M return a 404 for robots.txt, against 4.9% in the top 1K. They also serve HTML in place of the file more often: 5.5% against 3.2% in the top 1K.
Top sites fail differently. They are the most likely to put the file behind a login or firewall: 2.5% of the 7,500 origins ranked 1 to 10K return a 401 or 403, and so do 2.5% of origins ranked 10K to 50K, against 0.54% of origins beyond 1M. That pattern usually comes from bot protection or a web application firewall that challenges unknown clients, and robots.txt gets caught in the same rule as everything else.
The fix differs too. A missing file needs to be created. A blocked file needs an exception in the firewall or bot rules so crawlers can always read it.
Big sites write longer files, and a few are far too big
Among working files, the most popular sites write much longer ones. The median working file in the top 1K is about 1,347 bytes, against about 200 bytes for origins ranked beyond 1M. Popular sites tend to have more sections to exclude, more crawlers to address and more sitemaps to list.
At the extreme, some responses are far too large. 0.12% of responses (18,976) exceed the 500 KiB limit Google documents for processing robots.txt, and the largest is 47.1 MiB. But only 2,108 of those oversized responses are working text files. The rest are HTML pages or error responses, which suggests that most oversized "robots.txt" files are really a full page sent from the wrong path.
What this means for your site
Request /robots.txt on every host you run, including subdomains, and check three things:
- It returns a 200 with a text content type, not an HTML page and not a redirect to the homepage.
- It is reachable without a login, cookie or challenge page. If you use a firewall or bot protection, add an explicit allow rule for /robots.txt.
- It stays well under 500 KiB, and every line is a rule you meant to write.
If you have no file and want crawlers to see everything, a short file that only lists your sitemap is enough. Then check the robots.txt report in Search Console, which shows the last version Google fetched and any errors it hit. Watch for server errors in particular, because Google documents that it may slow or pause crawling while robots.txt returns one.
Sources
- HTTP Archive crawl, September 2026, robots.txt responses for all 15.5M origins
- Google Search Central, how Google interprets the robots.txt specification: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
Embed this research
Paste this on your site to embed the chart. It links back to the source automatically.
<iframe id="psm-robots-txt-statistics" src="https://www.pagespeedmatters.com/embed/data-studies/robots-txt-statistics" width="100%" height="600" style="width:100%;border:1px solid #E2E3E1;border-radius:12px" loading="lazy" title="Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File - PageSpeed Matters"></iframe>
<script>window.addEventListener("message",function(e){if(e&&e.data&&e.data.psmEmbed==="robots-txt-statistics"&&e.data.height){var f=document.getElementById("psm-robots-txt-statistics");if(f){f.style.height=e.data.height+"px";}}});</script>
<p style="font:14px/1.5 system-ui,sans-serif">Source: <a href="https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics">Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File - PageSpeed Matters</a></p>
Frequently asked questions
What percentage of websites have a robots.txt file?
Most do. In the September 2026 HTTP Archive crawl, 17.5% of 15,470,478 origins, about 1 in 6, had no working robots.txt. 11.75% returned a 404 or 410 and 3.91% returned an HTML page instead of a text file.
Which sites are most likely to have a broken robots.txt?
Mid-size sites. 22.5% of origins ranked 50K to 1M by Chrome popularity lack a working file, peaking at 23.0% in the 100K to 500K bucket, against 10.9% of the top 1K.
How big can a robots.txt file be?
Google documents a 500 KiB processing limit. In our data, 0.12% of responses (18,976) exceeded it, but only 2,108 of those were working text files; the rest were HTML pages or error responses.
How do I check my robots.txt?
Request /robots.txt on each host and confirm it returns a 200 as a text file, without a login or challenge page. Search Console's robots.txt report shows the version Google last fetched and any errors.
A missing, blocked or misrouted robots.txt is usually a single server or firewall rule, and fixing it gives crawlers a clear set of instructions on every visit. Need help with this? See our technical SEO services.