Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File

    17.5% of the 15,470,478 origins in the September 2026 HTTP Archive crawl, about 1 in 6, have no working robots.txt file.

    Share of sites without a working robots.txt, by Chrome popularity rank
    Top 1K10.9%
    1K to 5K14.6%
    5K to 10K16.2%
    10K to 50K20.8%
    50K to 100K22.6%
    100K to 500K23%
    500K to 1M22.2%
    1M to 5M18.8%
    5M to 10M16.3%
    Beyond 10M17.1%

    Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, all 15.5M origins.

    Server errors can be temporary, so a single crawl may catch some sites on a bad day.

    Source: Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File - PageSpeed Matters