---
title: "AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% - PageSpeed Matters"
url: https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt
description: "11.8% of 13,358,406 working robots.txt files name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot."
lang: en
---

AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% (https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt)

11.8% of 13,358,406 working robots.txt files name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot.

Robots.txt files naming GPTBot vs Googlebot, by site popularity

GPTBot, Top 10K 23.8%

Googlebot, Top 10K 8.8%

GPTBot, rest of Top 100K 19.3%

Googlebot, rest of Top 100K 8.4%

GPTBot, 100,001 to 1M 15.2%

Googlebot, 100,001 to 1M 7.5%

GPTBot, beyond 1M 11.6%

Googlebot, beyond 1M 5.2%

Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, 13,358,406 files returning a 200.

Naming a crawler is not the same as blocking it. Tiers exclude the more popular tiers above them.

Source: AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% - PageSpeed Matters (https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt)