Share of robots.txt files naming each crawler, by site popularity
| Crawler | Top 10K | Rest of Top 100K | 100,001 to 1M | Beyond top 1M | All files |
|---|---|---|---|---|---|
| GPTBot | 23.8% | 19.3% | 15.2% | 11.6% | 11.8% |
| ClaudeBot | 21.4% | 17.7% | 14.3% | 10.7% | 10.9% |
| CCBot | 21.3% | 17.2% | 13.5% | 9.9% | 10.1% |
| Google-Extended | 19.2% | 15.9% | 12.7% | 9.7% | 9.8% |
| OAI-SearchBot or ChatGPT-User | 12.3% | 8.5% | 6.0% | 3.8% | 4.0% |
| PerplexityBot | 11.6% | 7.5% | 5.2% | 3.4% | 3.5% |
| Googlebot | 8.8% | 8.4% | 7.5% | 5.2% | 5.3% |
| Bingbot | 8.1% | 5.9% | 4.4% | 3.6% | 3.6% |
| Files in tier | 6,620 | 56,241 | 587,271 | 12,708,274 | 13,358,406 |
Source: PageSpeed Matters, HTTP Archive Sept 2026. Counts files with a user-agent group for each crawler, whatever the rules say.
Summary
Robots.txt was built for search engines, but many files now also carry rules for AI crawlers: the bots that collect training data and the agents that fetch pages for AI answers. We wanted to know how often site owners write rules for these crawlers by name, and how that compares with Googlebot.
Across 13,358,406 robots.txt files that return a 200, 11.8% name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot. Site owners are more than twice as likely to address an AI training crawler by name as Google's own search crawler, and the most popular sites do it most.
What we measured
We looked at the robots.txt files for all origins in the September 2026 HTTP Archive crawl, and kept only files that returned a 200: 13,358,406 in all.
A crawler counts as named when its token appears as a user-agent value in the file, matched exactly and ignoring case. This measures whether a file has a group for that crawler, not what the group says. A group can block a crawler, allow it, or limit it to some folders, so these figures do not show how many sites block AI crawlers.
We also counted disallow rules in the Googlebot group and the catch-all * group. Popularity tiers use each origin's Chrome UX Report rank: Top 10K, the rest of the Top 100K, ranks 100,001 to 1M, and beyond the top 1M.
More files name GPTBot than Googlebot
Of the 13,358,406 robots.txt files that return a 200, 11.8% (1,571,805) name GPTBot, OpenAI's crawler, and 10.9% (1,456,106) name ClaudeBot, Anthropic's crawler. Only 5.3% (711,228) name Googlebot.
The gap likely has a simple cause. Most sites are happy for Googlebot to crawl everything, so they cover it with the * group and never name it. AI crawlers are new, and site owners who have a view on them have to write a specific group to express it.
Google-Extended, the token that controls Google's AI training use, is named in 9.8% of valid files (1,312,292), nearly twice the share that name Googlebot itself. Google-Extended does not affect how Googlebot crawls or ranks a site for Search, so naming it is a way to set an AI training preference while leaving search crawling alone.
11.8%
Share of 13,358,406 robots.txt files returning a 200 that name GPTBot in a user-agent group.
Popular sites name AI crawlers more often
The more popular the site, the more likely its robots.txt names AI crawlers. 23.8% of 6,620 Top 10K files name GPTBot, against 15.2% of 587,271 files ranked 100,001 to 1M and 11.6% of 12.7M files beyond the top 1M.
Among Top 10K files, 21.4% name ClaudeBot and 21.3% name CCBot, Common Crawl's crawler, against 8.8% naming Googlebot and 8.1% naming Bingbot. Popular sites likely have more of the content AI companies want, and more often have teams that set a policy on it.
Search-focused AI agents trail the training crawlers. OAI-SearchBot or ChatGPT-User appear in 4.0% of files and PerplexityBot in 3.5%, rising to 12.3% and 11.6% among Top 10K files. These agents fetch pages to answer questions and can send visitors back, which likely explains why fewer sites write rules for them.
Googlebot groups loosen rules more often than they tighten them
676,853 files have both a Googlebot group and a group. Googlebot-specific groups loosen rules more often than they tighten them. 26.4% of those files (178,614) give Googlebot no disallows at all while the group has some.
Counting only files where both groups have at least one disallow, 20.4% give Googlebot fewer disallows than the * group and 9.9% give it more. In other words, when site owners write a separate group for Google, it is far more often to let Google in where other bots are kept out than to keep Google out.
One detail is easy to miss. A crawler follows only the most specific group that names it, so a Googlebot group replaces the group for Google rather than adding to it. Any rule in the group that should also apply to Google has to be repeated in the Googlebot group.
What this means for your site
Decide what you want from each type of crawler before you edit the file.
- To keep search traffic, leave Googlebot and Bingbot able to reach the pages you want ranked.
- To limit AI training use, write groups for the training crawlers you care about, such as GPTBot, ClaudeBot, CCBot and Google-Extended.
- Think separately about search-focused agents like OAI-SearchBot and PerplexityBot, which fetch pages to answer questions.
- If you add a Googlebot group, copy over every * rule that should still apply to Google.
Then check the file in Search Console's robots.txt report to confirm Google reads it the way you intend.
Sources
- HTTP Archive crawl, September 2026, robots.txt files for all origins
- Google Search Central, Overview of Google crawlers and fetchers: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- Google Search Central, How Google interprets the robots.txt specification: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
Embed this research
Paste this on your site to embed the chart. It links back to the source automatically.
<iframe id="psm-ai-crawlers-in-robots-txt" src="https://www.pagespeedmatters.com/embed/data-studies/ai-crawlers-in-robots-txt" width="100%" height="600" style="width:100%;border:1px solid #E2E3E1;border-radius:12px" loading="lazy" title="AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% - PageSpeed Matters"></iframe>
<script>window.addEventListener("message",function(e){if(e&&e.data&&e.data.psmEmbed==="ai-crawlers-in-robots-txt"&&e.data.height){var f=document.getElementById("psm-ai-crawlers-in-robots-txt");if(f){f.style.height=e.data.height+"px";}}});</script>
<p style="font:14px/1.5 system-ui,sans-serif">Source: <a href="https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt">AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% - PageSpeed Matters</a></p>
Frequently asked questions
How many websites mention GPTBot in robots.txt?
11.8% of 13,358,406 robots.txt files returning a 200 name GPTBot in a user-agent group, rising to 23.8% among Top 10K sites. That counts files with a rule for GPTBot, not only those that block it.
Do more sites name AI crawlers than Googlebot?
Yes. 11.8% of files name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot. Google-Extended, Google's AI training token, is named in 9.8%.
Does adding a Googlebot group to robots.txt change the * rules for Google?
Yes, Googlebot follows its own group instead of the * group. Of 676,853 files with both, 26.4% give Googlebot no disallows while the * group has some.
How do I check which crawlers my robots.txt addresses?
Open yoursite.com/robots.txt and read each user-agent line. Every crawler named there follows only its own group, and every other crawler follows the * group.
A clear robots.txt policy lets you keep search crawlers working for you while deciding on your own terms how AI crawlers treat your content. Need help with this? See our technical SEO services.