---
title: "Robots.txt Mistakes: 21.7% Use Rules Google Ignores - PageSpeed Matters"
url: https://www.pagespeedmatters.com/resources/data-studies/robots-txt-mistakes
description: "21.7% of 13,358,406 working robots.txt files contain at least one directive Google doesn't support."
lang: en
---

Robots.txt Mistakes: 21.7% Use Rules Google Ignores (https://www.pagespeedmatters.com/resources/data-studies/robots-txt-mistakes)

21.7% of 13,358,406 working robots.txt files contain at least one directive Google doesn't support.

Robots.txt files with an unsupported directive, by number of user-agent groups

No user-agent line 51%

1 group 9.7%

2 to 5 groups 20.4%

6 to 20 groups 52.3%

21+ groups 25.8%

Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, 13,358,406 files returning a 200.

Many files with no user-agent line are likely HTML pages served at the robots.txt address.

Source: Robots.txt Mistakes: 21.7% Use Rules Google Ignores - PageSpeed Matters (https://www.pagespeedmatters.com/resources/data-studies/robots-txt-mistakes)