---
title: "Robots.txt Statistics 2026: 17.5% Lack a Working File"
url: https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics
description: "17.5% of 15,470,478 origins have no working robots.txt: 11.75% return a 404 or 410 and 3.91% serve HTML. Sites ranked 50K to 1M fare worst at 22.5%."
lang: en
---
Data Study ยท Technical SEO
# Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File
17.5% of the 15,470,478 origins in the September 2026 HTTP Archive crawl, about 1 in 6, have no working robots.txt file.
- 15,470,478 origins
- HTTP Archive, Sept 2026
Matt Suffoletto: https://www.pagespeedmatters.com/resources/author/matt-suffoletto
Founder, PageSpeed Matters Published Oct. 19, 2026 7 min read
## Key Findings
1. Across all 15,470,478 origins in the September 2026 HTTP Archive crawl, **17.5%** have no working robots.txt: **11.75%** return a 404 or 410, 3.91% return an HTML page, 0.58% block it with a 401 or 403, 0.25% return a 429 or server error and 0.99% don't respond. Copy stat
2. Mid-size sites fare worst: **22.5%** of the 745,671 origins ranked 50K to 1M lack a working file, against **10.9%** of the top 1K and 17.3% of the 14.69M origins ranked beyond 1M. Copy stat
3. The worst single rank bucket is 100K to 500K, where **23.0%** of 309,944 origins have no working robots.txt. Copy stat
4. Mid-size sites mostly fail by having no file at all: **14.4%** of origins ranked 50K to 1M return a 404, against **4.9%** in the top 1K. Copy stat
5. Top sites are the most likely to put the file behind a login or firewall: **2.5%** of the 7,500 origins ranked 1 to 10K return a 401 or 403, against 0.54% of origins beyond 1M. Copy stat
6. Top 1K sites write much longer files, with a median working file of about **1,347 bytes** against about **200 bytes** for origins ranked beyond 1M. Copy stat
7. **0.12%** of responses (18,976) exceed Google's 500 KiB processing limit and the largest is 47.1 MiB, but only **2,108** of those oversized responses are working text files; the rest are HTML or error responses. Copy stat
Share of sites without a working robots.txt, by Chrome popularity rank
Top 1K 10.9%
1K to 5K 14.6%
5K to 10K 16.2%
10K to 50K 20.8%
50K to 100K 22.6%
100K to 500K 23%
500K to 1M 22.2%
1M to 5M 18.8%
5M to 10M 16.3%
Beyond 10M 17.1%
Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, all 15.5M origins.
Server errors can be temporary, so a single crawl may catch some sites on a bad day.
In this study
## Sites without a working robots.txt by popularity (HTTP Archive, Sept 2026)
| Chrome rank | Origins | No file (404/410) | HTML page | 401/403 | 429/5xx | No response | Not working total | Median working file (bytes) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Top 1K | 754 | 4.9% | 3.2% | 1.5% | 0.1% | 1.2% | 10.9% | 1,347 |
| 1K to 10K | 6,746 | 8.2% | 3.4% | 2.6% | 0.1% | 1.2% | 15.5% | n/a |
| 10K to 50K | 29,679 | 11.7% | 5.1% | 2.5% | 0.2% | 1.3% | 20.8% | n/a |
| 50K to 1M | 745,671 | 14.4% | 5.5% | 1.3% | 0.2% | 1.0% | 22.5% | 411 |
| Beyond 1M | 14,687,628 | 11.6% | 3.8% | 0.5% | 0.3% | 1.0% | 17.3% | 200 |
| All | 15,470,478 | 11.7% | 3.9% | 0.6% | 0.25% | 1.0% | 17.5% | 206 |
Source: PageSpeed Matters, Sept 2026. Medians are approximate; n/a means no median was reported for that bucket.
## Summary
A robots.txt file is the first thing a crawler asks for on a site. It tells search engines which paths to skip and often where the sitemap lives. When the request fails, crawlers have to fall back on their own defaults, and the site loses that bit of control.
We checked the robots.txt response for every origin in the September 2026 HTTP Archive crawl. Of 15,470,478 origins, 17.5%, about 1 in 6, have no working robots.txt file. The failure rate is highest not on tiny sites but on mid-size ones ranked 50K to 1M by Chrome popularity.
## What we measured
We looked at the robots.txt response for every origin in the September 2026 HTTP Archive crawl, all 15.5M of them, and grouped origins by their Chrome popularity rank.
A file counts as working when the server returns it as a text file. Everything else counts as not working, split into five groups: a 404 or 410 (no file), an HTML page served instead of a text file, a 401 or 403 (blocked by a login or firewall), a 429 or server error, and no response at all.
Two limits apply. HTML responses are usually catch-all pages, not deliberate blocks, and server errors can be temporary, so a single crawl may catch a site on a bad day. File size medians are approximate.
## One in six origins has no working robots.txt
Across all 15,470,478 origins, 17.5% have no working robots.txt. Most of those simply have no file: 11.75% return a 404 or 410.
The rest break in other ways. 3.91% return an HTML page instead of a text file, usually a homepage or error template that the server sends for any unknown path. 0.58% block the file with a 401 or 403. 0.25% return a 429 or a server error, and 0.99% don't respond.
These failures are not equal. A missing file is the mildest case: Google's documentation says it treats most 4xx responses as if there were no robots.txt, so it crawls without restrictions. An HTML page in place of the file, or a file that sits behind a server error, is messier, because the site has lost a clear signal it could easily send.
17.5%
Share of the 15,470,478 origins in the September 2026 HTTP Archive crawl with no working robots.txt file.
## Mid-size sites fare worst
The failure rate rises as sites get less popular, peaks in the middle and then eases off again. 10.9% of the top 1K origins lack a working file. That rises to 22.5% of the 745,671 origins ranked 50K to 1M, then drops back to 17.3% of the 14.69M origins ranked beyond 1M.
The worst single bucket is 100K to 500K, where 23.0% of 309,944 origins have no working robots.txt. The chart shows the curve: 14.6% for 1K to 5K, 16.2% for 5K to 10K, 20.8% for 10K to 50K, then 22.6%, 23.0% and 22.2% through the 50K to 1M range.
Our data does not show why, but a likely reason is that mid-size sites are big enough to run custom stacks and many hosts, while often lacking someone whose job is to check the basics on each one.
## How sites fail depends on their size
Mid-size sites mostly fail by having no file at all. 14.4% of origins ranked 50K to 1M return a 404 for robots.txt, against 4.9% in the top 1K. They also serve HTML in place of the file more often: 5.5% against 3.2% in the top 1K.
Top sites fail differently. They are the most likely to put the file behind a login or firewall: 2.5% of the 7,500 origins ranked 1 to 10K return a 401 or 403, and so do 2.5% of origins ranked 10K to 50K, against 0.54% of origins beyond 1M. That pattern usually comes from bot protection or a web application firewall that challenges unknown clients, and robots.txt gets caught in the same rule as everything else.
The fix differs too. A missing file needs to be created. A blocked file needs an exception in the firewall or bot rules so crawlers can always read it.
## Big sites write longer files, and a few are far too big
Among working files, the most popular sites write much longer ones. The median working file in the top 1K is about 1,347 bytes, against about 200 bytes for origins ranked beyond 1M. Popular sites tend to have more sections to exclude, more crawlers to address and more sitemaps to list.
At the extreme, some responses are far too large. 0.12% of responses (18,976) exceed the 500 KiB limit Google documents for processing robots.txt, and the largest is 47.1 MiB. But only 2,108 of those oversized responses are working text files. The rest are HTML pages or error responses, which suggests that most oversized "robots.txt" files are really a full page sent from the wrong path.
## What this means for your site
Request /robots.txt on every host you run, including subdomains, and check three things:
- It returns a 200 with a text content type, not an HTML page and not a redirect to the homepage.
- It is reachable without a login, cookie or challenge page. If you use a firewall or bot protection, add an explicit allow rule for /robots.txt.
- It stays well under 500 KiB, and every line is a rule you meant to write.
If you have no file and want crawlers to see everything, a short file that only lists your sitemap is enough. Then check the robots.txt report in Search Console, which shows the last version Google fetched and any errors it hit. Watch for server errors in particular, because Google documents that it may slow or pause crawling while robots.txt returns one.
## Sources
- HTTP Archive crawl, September 2026, robots.txt responses for all 15.5M origins
- Google Search Central, how Google interprets the robots.txt specification: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
## Embed this research
Paste this on your site to embed the chart. It links back to the source automatically.
Source: Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File - PageSpeed Matters
## Frequently asked questions
### What percentage of websites have a robots.txt file?
Most do. In the September 2026 HTTP Archive crawl, 17.5% of 15,470,478 origins, about 1 in 6, had no working robots.txt. 11.75% returned a 404 or 410 and 3.91% returned an HTML page instead of a text file.
### Which sites are most likely to have a broken robots.txt?
Mid-size sites. 22.5% of origins ranked 50K to 1M by Chrome popularity lack a working file, peaking at 23.0% in the 100K to 500K bucket, against 10.9% of the top 1K.
### How big can a robots.txt file be?
Google documents a 500 KiB processing limit. In our data, 0.12% of responses (18,976) exceeded it, but only 2,108 of those were working text files; the rest were HTML pages or error responses.
### How do I check my robots.txt?
Request /robots.txt on each host and confirm it returns a 200 as a text file, without a login or challenge page. Search Console's robots.txt report shows the version Google last fetched and any errors.
A missing, blocked or misrouted robots.txt is usually a single server or firewall rule, and fixing it gives crawlers a clear set of instructions on every visit. Need help with this? See our technical SEO services (https://www.pagespeedmatters.com/services/technical-seo).
Share
How we measured
## Methodology & Source
Data source
HTTP Archive crawl, September 2026: robots.txt response for every crawled origin
Collection date
September 2026
Sample
15,470,478 origins
Release
HTTP Archive, Sept 2026
All 15.5M origins in the September 2026 HTTP Archive crawl, grouped by Chrome popularity rank. A working file is one returned as a text file; 404/410, HTML pages, 401/403, 429/5xx and no response all count as not working. HTML responses are usually catch-all pages rather than deliberate blocks, and 5xx errors can be temporary. Medians are approximate.
Using this data?
## Cite this study
Link back to the study when you cite it.
APA
Matt Suffoletto. (2026). Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File. PageSpeed Matters. https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics
Plain link
Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File - PageSpeed Matters - https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics
## Related studies
- Technical SEO Robots.txt Mistakes: 21.7% Use Rules Google Ignores: https://www.pagespeedmatters.com/resources/data-studies/robots-txt-mistakes
- Technical SEO AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%: https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt
- Technical SEO XML Sitemap Errors: 1 in 4 Sampled Sites List URLs Google Shouldn't Index: https://www.pagespeedmatters.com/resources/data-studies/xml-sitemap-errors
- Technical SEO Soft 404s on Ranking Sites: 8.3% Serve Pages for Fake URLs: https://www.pagespeedmatters.com/resources/data-studies/soft-404-statistics
## Ready to Get Started?
Get your site optimized in as little as 3 days. From $500.
See Plans & Pricing: https://www.pagespeedmatters.com/pricing
## Structured data
```json
[
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File",
"datePublished": "2026-10-19",
"dateModified": "2026-10-19",
"author": {
"@type": "Person",
"name": "Matt Suffoletto",
"jobTitle": "Performance Optimization Expert",
"url": "https://www.pagespeedmatters.com/resources/author/matt-suffoletto"
}
},
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://www.pagespeedmatters.com/"
},
{
"@type": "ListItem",
"position": 2,
"name": "Resources",
"item": "https://www.pagespeedmatters.com/resources"
},
{
"@type": "ListItem",
"position": 3,
"name": "Data Studies",
"item": "https://www.pagespeedmatters.com/resources/data-studies"
},
{
"@type": "ListItem",
"position": 4,
"name": "Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File"
}
]
},
{
"@context": "https://schema.org",
"@type": "Dataset",
"name": "Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File",
"description": "17.5% of 15,470,478 origins have no working robots.txt: 11.75% return a 404 or 410 and 3.91% serve HTML. Sites ranked 50K to 1M fare worst at 22.5%.",
"url": "https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics",
"creator": {
"@type": "Organization",
"name": "PageSpeed Matters",
"url": "https://www.pagespeedmatters.com"
},
"datePublished": "2026-10-19",
"dateModified": "2026-10-19",
"license": "https://creativecommons.org/licenses/by/4.0/",
"isAccessibleForFree": true,
"creativeWorkStatus": "Published",
"variableMeasured": "Core Web Vitals pass rate and web performance metrics",
"measurementTechnique": "HTTP Archive crawl, September 2026: robots.txt response for every crawled origin"
},
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What percentage of websites have a robots.txt file?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Most do. In the September 2026 HTTP Archive crawl, 17.5% of 15,470,478 origins, about 1 in 6, had no working robots.txt. 11.75% returned a 404 or 410 and 3.91% returned an HTML page instead of a text file."
}
},
{
"@type": "Question",
"name": "Which sites are most likely to have a broken robots.txt?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Mid-size sites. 22.5% of origins ranked 50K to 1M by Chrome popularity lack a working file, peaking at 23.0% in the 100K to 500K bucket, against 10.9% of the top 1K."
}
},
{
"@type": "Question",
"name": "How big can a robots.txt file be?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Google documents a 500 KiB processing limit. In our data, 0.12% of responses (18,976) exceeded it, but only 2,108 of those were working text files; the rest were HTML pages or error responses."
}
},
{
"@type": "Question",
"name": "How do I check my robots.txt?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Request /robots.txt on each host and confirm it returns a 200 as a text file, without a login or challenge page. Search Console's robots.txt report shows the version Google last fetched and any errors."
}
}
]
}
]
```