Robots.txt Mistakes: 21.7% Use Rules Google Ignores

    21.7% of 13,358,406 working robots.txt files contain at least one directive Google doesn't support.

    Robots.txt files with an unsupported directive, by number of user-agent groups
    No user-agent line51%
    1 group9.7%
    2 to 5 groups20.4%
    6 to 20 groups52.3%
    21+ groups25.8%

    Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, 13,358,406 files returning a 200.

    Many files with no user-agent line are likely HTML pages served at the robots.txt address.

    Source: Robots.txt Mistakes: 21.7% Use Rules Google Ignores - PageSpeed Matters