AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%

    11.8% of 13,358,406 working robots.txt files name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot.

    Robots.txt files naming GPTBot vs Googlebot, by site popularity
    GPTBot, Top 10K23.8%
    Googlebot, Top 10K8.8%
    GPTBot, rest of Top 100K19.3%
    Googlebot, rest of Top 100K8.4%
    GPTBot, 100,001 to 1M15.2%
    Googlebot, 100,001 to 1M7.5%
    GPTBot, beyond 1M11.6%
    Googlebot, beyond 1M5.2%

    Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, 13,358,406 files returning a 200.

    Naming a crawler is not the same as blocking it. Tiers exclude the more popular tiers above them.

    Source: AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% - PageSpeed Matters