Robots.txt files with directives Google ignores, by site popularity
| Tier | Any unsupported | Crawl-delay | Host | Unrecognized lines | Content-signal | Files |
|---|---|---|---|---|---|---|
| Top 10K | 21.7% | 11.3% | 3.7% | 5.9% | 8.6% | 6,620 |
| Ranked 10,001 to 100K | 22.5% | 10.3% | 3.5% | 8.7% | 9.5% | 56,241 |
| Ranked 100,001 to 1M | 22.4% | 10.7% | 2.9% | 8.8% | 8.5% | 587,271 |
| Beyond top 1M | 21.6% | 13.6% | 2.6% | 5.4% | 5.7% | 12,708,274 |
| All valid files | 21.7% | 13.5% | 2.7% | 5.5% | 5.8% | 13,358,406 |
Source: PageSpeed Matters, HTTP Archive Sept 2026. Top 10K includes the Top 1K. Content-signal lines are not counted as unsupported.
Summary
Google reads only a short list of robots.txt rules: user-agent, allow, disallow and sitemap. Anything else in the file is ignored by Googlebot. That is not always a problem, since some lines are written for other crawlers. But site owners often add rules like noindex or crawl-delay expecting Google to follow them, and it does not.
We checked every robots.txt file that returned a 200 in the September 2026 HTTP Archive crawl. 21.7% of 13,358,406 files that return a 200 contain at least one directive Google doesn't support. Crawl-delay is by far the most common, and the files most likely to carry these rules are the ones with a mid-sized set of crawler groups.
What we measured
We looked at the robots.txt files for all origins in the September 2026 HTTP Archive crawl. We kept only files that returned a 200, which leaves 13,358,406 files, and counted rules by directive type.
A file counts as having an unsupported directive when it contains any of crawl-delay, host, noindex, clean-param, request-rate, visit-time or an unrecognized line. Popularity tiers use each origin's Chrome UX Report rank. The Top 10K tier includes the Top 1K, and the other tiers cover ranks 10,001 to 100,000, ranks 100,001 to 1M, and everything beyond the top 1M.
One limit to keep in mind: unsupported by Google does not mean wrong. Crawl-delay still matters to Bing and other crawlers, so a crawl-delay line is a mistake only if you expected Google to obey it.
One in five robots.txt files has a rule Google ignores
2,894,671 of the 13,358,406 robots.txt files that return a 200 (21.7%) contain at least one directive Google doesn't support.
Crawl-delay is the most common, in 13.5% of files (1,800,570). Next come unrecognized lines in 5.5% (740,390), which are usually typos, stray text or rules in a format no crawler expects. Host appears in 2.7% (354,625) and noindex in 46,223 files. Noindex is the riskiest of these: a site owner who puts it in robots.txt expecting a page to drop out of Google's index may find it still indexed.
Separately, 5.8% of files (776,018) now include content-signal lines, a newer convention for stating how content may be used by AI systems. These are not counted in the 21.7%.
21.7%
Share of 13,358,406 robots.txt files returning a 200 that contain at least one directive Google doesn't support.
Mid-sized files are the most error-prone
The number of user-agent groups in a file is the strongest pattern in the data. 52.3% of 1,871,323 files with 6 to 20 groups carry an unsupported directive, against 9.7% of 7,393,897 files with a single group. Files with 2 to 5 groups sit at 20.4%, and files with 21 or more groups fall back to 25.8%.
A likely reason is how these files are built. A file with one group is usually a short default. A file with 6 to 20 groups is often assembled by hand or from copied templates, with separate blocks for different bots and a crawl-delay added to several of them.
8.1% of valid files (1,076,704) have no user-agent line at all, so their rules apply to no crawler. 541,617 of these are served with a content type other than text/plain, which points to an HTML page at the robots.txt address, such as a homepage or error page returned with a 200. 51.0% of files with no user-agent line also contain an unsupported directive, which fits a page that is not really a robots.txt file.
Popular sites use crawl-delay less and content signals more
Crawl-delay is less common among popular sites. 11.3% of 6,620 Top 10K files and 10.7% of 587,271 files ranked 100,001 to 1M use it, against 13.6% of the 12.7M files from origins beyond the top 1M. The overall share of files with any unsupported directive barely moves across tiers, from 21.6% beyond the top 1M to 22.5% for ranks 10,001 to 100,000.
Content-signal lines are more common among ranked sites. 8.6% of Top 10K files and 9.5% of 56,241 files ranked 10,001 to 100,000 include them, against 5.7% beyond the top 1M.
Clean-param, a directive only Yandex reads, appears in 135,788 files (1.0%) and in 2.2% of Top 10K files. It may be deliberate on sites that target Yandex users; elsewhere it is likely left over from a template.
What this means for your site
Open yoursite.com/robots.txt in a browser and read it line by line.
- If it shows an HTML page instead of plain text, fix the server so the address returns a real robots.txt file or a 404.
- Remove noindex lines. To keep a page out of Google, use a noindex meta tag or an X-Robots-Tag header on a page Google can crawl.
- Keep crawl-delay only if you need it for Bing or other crawlers, and know that Google ignores it.
- Check that every block starts with a user-agent line, and fix typos that turn rules into unrecognized lines.
Then run the file through the robots.txt report in Search Console to see how Google reads it.
Sources
- HTTP Archive crawl, September 2026, robots.txt files for all origins
- Google Search Central, How Google interprets the robots.txt specification: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
Embed this research
Paste this on your site to embed the chart. It links back to the source automatically.
<iframe id="psm-robots-txt-mistakes" src="https://www.pagespeedmatters.com/embed/data-studies/robots-txt-mistakes" width="100%" height="600" style="width:100%;border:1px solid #E2E3E1;border-radius:12px" loading="lazy" title="Robots.txt Mistakes: 21.7% Use Rules Google Ignores - PageSpeed Matters"></iframe>
<script>window.addEventListener("message",function(e){if(e&&e.data&&e.data.psmEmbed==="robots-txt-mistakes"&&e.data.height){var f=document.getElementById("psm-robots-txt-mistakes");if(f){f.style.height=e.data.height+"px";}}});</script>
<p style="font:14px/1.5 system-ui,sans-serif">Source: <a href="https://www.pagespeedmatters.com/resources/data-studies/robots-txt-mistakes">Robots.txt Mistakes: 21.7% Use Rules Google Ignores - PageSpeed Matters</a></p>
Frequently asked questions
Does Google support crawl-delay in robots.txt?
No. Google ignores crawl-delay, though Bing and other crawlers still read it. It is the most common unsupported directive, in 13.5% of the 13,358,406 robots.txt files returning a 200 we checked.
How common are robots.txt mistakes?
21.7% of 13,358,406 robots.txt files that return a 200 contain at least one directive Google doesn't support, such as crawl-delay, host, noindex or unrecognized lines. 8.1% have no user-agent line at all.
Can I use noindex in robots.txt?
Not for Google, which does not support it. 46,223 robots.txt files returning a 200 still contain a noindex line. Use a noindex meta tag or X-Robots-Tag header instead.
How do I check my robots.txt file?
Open it in a browser and confirm it is plain text with a user-agent line before each block of rules. Then check Search Console's robots.txt report to see how Google reads it.
A robots.txt file is short enough to clean up in one pass, and getting it right means crawlers follow the rules you actually meant to set. Need help with this? See our technical SEO services.