---
title: "Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File - PageSpeed Matters"
url: https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics
description: "17.5% of the 15,470,478 origins in the September 2026 HTTP Archive crawl, about 1 in 6, have no working robots.txt file."
lang: en
---

Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File (https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics)

17.5% of the 15,470,478 origins in the September 2026 HTTP Archive crawl, about 1 in 6, have no working robots.txt file.

Share of sites without a working robots.txt, by Chrome popularity rank

Top 1K 10.9%

1K to 5K 14.6%

5K to 10K 16.2%

10K to 50K 20.8%

50K to 100K 22.6%

100K to 500K 23%

500K to 1M 22.2%

1M to 5M 18.8%

5M to 10M 16.3%

Beyond 10M 17.1%

Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, all 15.5M origins.

Server errors can be temporary, so a single crawl may catch some sites on a bad day.

Source: Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File - PageSpeed Matters (https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics)