---
title: "AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%"
url: https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt
description: "Of 13,358,406 robots.txt files returning a 200, 11.8% name GPTBot, 10.9% ClaudeBot and 5.3% Googlebot. Among Top 10K sites, 23.8% name GPTBot."
lang: en
---
Data Study ยท Technical SEO
# AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%
11.8% of 13,358,406 working robots.txt files name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot.
- 13,358,406 robots.txt files
- HTTP Archive, Sept 2026
Matt Suffoletto: https://www.pagespeedmatters.com/resources/author/matt-suffoletto
Founder, PageSpeed Matters Published Oct. 19, 2026 7 min read
## Key Findings
1. Of 13,358,406 robots.txt files that return a 200, **11.8%** (1,571,805) name GPTBot, 10.9% (1,456,106) name ClaudeBot and **5.3%** (711,228) name Googlebot. Copy stat
2. Google-Extended, the token that controls Google's AI training use, is named in **9.8%** of valid files (1,312,292), nearly twice the share that name Googlebot itself. Copy stat
3. The more popular the site, the more likely it names AI crawlers: **23.8%** of 6,620 Top 10K files name GPTBot, against 15.2% of 587,271 files ranked 100,001 to 1M and 11.6% of 12.7M files beyond the top 1M. Copy stat
4. Among Top 10K files, **21.4%** name ClaudeBot and 21.3% CCBot, against **8.8%** naming Googlebot and 8.1% naming Bingbot. Copy stat
5. Search-focused AI agents trail the training crawlers: OAI-SearchBot or ChatGPT-User appear in **4.0%** of files and PerplexityBot in 3.5%, rising to 12.3% and 11.6% among Top 10K files. Copy stat
6. Of 676,853 files with both a Googlebot and a * group, **20.4%** give Googlebot fewer disallows and **9.9%** give it more, counting files where both groups have at least one disallow. Copy stat
7. Googlebot-specific groups often exist to loosen rules: **26.4%** of those 676,853 files (178,614) give Googlebot no disallows at all while the * group has some. Copy stat
Robots.txt files naming GPTBot vs Googlebot, by site popularity
GPTBot, Top 10K 23.8%
Googlebot, Top 10K 8.8%
GPTBot, rest of Top 100K 19.3%
Googlebot, rest of Top 100K 8.4%
GPTBot, 100,001 to 1M 15.2%
Googlebot, 100,001 to 1M 7.5%
GPTBot, beyond 1M 11.6%
Googlebot, beyond 1M 5.2%
Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, 13,358,406 files returning a 200.
Naming a crawler is not the same as blocking it. Tiers exclude the more popular tiers above them.
In this study
## Share of robots.txt files naming each crawler, by site popularity
| Crawler | Top 10K | Rest of Top 100K | 100,001 to 1M | Beyond top 1M | All files |
| --- | --- | --- | --- | --- | --- |
| GPTBot | 23.8% | 19.3% | 15.2% | 11.6% | 11.8% |
| ClaudeBot | 21.4% | 17.7% | 14.3% | 10.7% | 10.9% |
| CCBot | 21.3% | 17.2% | 13.5% | 9.9% | 10.1% |
| Google-Extended | 19.2% | 15.9% | 12.7% | 9.7% | 9.8% |
| OAI-SearchBot or ChatGPT-User | 12.3% | 8.5% | 6.0% | 3.8% | 4.0% |
| PerplexityBot | 11.6% | 7.5% | 5.2% | 3.4% | 3.5% |
| Googlebot | 8.8% | 8.4% | 7.5% | 5.2% | 5.3% |
| Bingbot | 8.1% | 5.9% | 4.4% | 3.6% | 3.6% |
| Files in tier | 6,620 | 56,241 | 587,271 | 12,708,274 | 13,358,406 |
Source: PageSpeed Matters, HTTP Archive Sept 2026. Counts files with a user-agent group for each crawler, whatever the rules say.
## Summary
Robots.txt was built for search engines, but many files now also carry rules for AI crawlers: the bots that collect training data and the agents that fetch pages for AI answers. We wanted to know how often site owners write rules for these crawlers by name, and how that compares with Googlebot.
Across 13,358,406 robots.txt files that return a 200, 11.8% name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot. Site owners are more than twice as likely to address an AI training crawler by name as Google's own search crawler, and the most popular sites do it most.
## What we measured
We looked at the robots.txt files for all origins in the September 2026 HTTP Archive crawl, and kept only files that returned a 200: 13,358,406 in all.
A crawler counts as named when its token appears as a user-agent value in the file, matched exactly and ignoring case. This measures whether a file has a group for that crawler, not what the group says. A group can block a crawler, allow it, or limit it to some folders, so these figures do not show how many sites block AI crawlers.
We also counted disallow rules in the Googlebot group and the catch-all * group. Popularity tiers use each origin's Chrome UX Report rank: Top 10K, the rest of the Top 100K, ranks 100,001 to 1M, and beyond the top 1M.
## More files name GPTBot than Googlebot
Of the 13,358,406 robots.txt files that return a 200, 11.8% (1,571,805) name GPTBot, OpenAI's crawler, and 10.9% (1,456,106) name ClaudeBot, Anthropic's crawler. Only 5.3% (711,228) name Googlebot.
The gap likely has a simple cause. Most sites are happy for Googlebot to crawl everything, so they cover it with the * group and never name it. AI crawlers are new, and site owners who have a view on them have to write a specific group to express it.
Google-Extended, the token that controls Google's AI training use, is named in 9.8% of valid files (1,312,292), nearly twice the share that name Googlebot itself. Google-Extended does not affect how Googlebot crawls or ranks a site for Search, so naming it is a way to set an AI training preference while leaving search crawling alone.
11.8%
Share of 13,358,406 robots.txt files returning a 200 that name GPTBot in a user-agent group.
## Popular sites name AI crawlers more often
The more popular the site, the more likely its robots.txt names AI crawlers. 23.8% of 6,620 Top 10K files name GPTBot, against 15.2% of 587,271 files ranked 100,001 to 1M and 11.6% of 12.7M files beyond the top 1M.
Among Top 10K files, 21.4% name ClaudeBot and 21.3% name CCBot, Common Crawl's crawler, against 8.8% naming Googlebot and 8.1% naming Bingbot. Popular sites likely have more of the content AI companies want, and more often have teams that set a policy on it.
Search-focused AI agents trail the training crawlers. OAI-SearchBot or ChatGPT-User appear in 4.0% of files and PerplexityBot in 3.5%, rising to 12.3% and 11.6% among Top 10K files. These agents fetch pages to answer questions and can send visitors back, which likely explains why fewer sites write rules for them.
## Googlebot groups loosen rules more often than they tighten them
676,853 files have both a Googlebot group and a **group. Googlebot-specific groups loosen rules more often than they tighten them. 26.4% of those files (178,614) give Googlebot no disallows at all while the** group has some.
Counting only files where both groups have at least one disallow, 20.4% give Googlebot fewer disallows than the * group and 9.9% give it more. In other words, when site owners write a separate group for Google, it is far more often to let Google in where other bots are kept out than to keep Google out.
One detail is easy to miss. A crawler follows only the most specific group that names it, so a Googlebot group replaces the **group for Google rather than adding to it. Any rule in the** group that should also apply to Google has to be repeated in the Googlebot group.
## What this means for your site
Decide what you want from each type of crawler before you edit the file.
- To keep search traffic, leave Googlebot and Bingbot able to reach the pages you want ranked.
- To limit AI training use, write groups for the training crawlers you care about, such as GPTBot, ClaudeBot, CCBot and Google-Extended.
- Think separately about search-focused agents like OAI-SearchBot and PerplexityBot, which fetch pages to answer questions.
- If you add a Googlebot group, copy over every * rule that should still apply to Google.
Then check the file in Search Console's robots.txt report to confirm Google reads it the way you intend.
## Sources
- HTTP Archive crawl, September 2026, robots.txt files for all origins
- Google Search Central, Overview of Google crawlers and fetchers: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- Google Search Central, How Google interprets the robots.txt specification: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
## Embed this research
Paste this on your site to embed the chart. It links back to the source automatically.
Source: AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% - PageSpeed Matters
## Frequently asked questions
### How many websites mention GPTBot in robots.txt?
11.8% of 13,358,406 robots.txt files returning a 200 name GPTBot in a user-agent group, rising to 23.8% among Top 10K sites. That counts files with a rule for GPTBot, not only those that block it.
### Do more sites name AI crawlers than Googlebot?
Yes. 11.8% of files name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot. Google-Extended, Google's AI training token, is named in 9.8%.
### Does adding a Googlebot group to robots.txt change the * rules for Google?
Yes, Googlebot follows its own group instead of the * group. Of 676,853 files with both, 26.4% give Googlebot no disallows while the * group has some.
### How do I check which crawlers my robots.txt addresses?
Open yoursite.com/robots.txt and read each user-agent line. Every crawler named there follows only its own group, and every other crawler follows the * group.
A clear robots.txt policy lets you keep search crawlers working for you while deciding on your own terms how AI crawlers treat your content. Need help with this? See our technical SEO services (https://www.pagespeedmatters.com/services/technical-seo).
Share
How we measured
## Methodology & Source
Data source
HTTP Archive crawl, September 2026: robots.txt files for all origins
Collection date
September 2026
Sample
13,358,406 robots.txt files
Release
HTTP Archive, Sept 2026
User-agent groups and disallow counts for robots.txt files returning a 200. A crawler counts as named when its token is a user-agent value; this shows that a file has a group for the crawler, not that it blocks it. The fewer and more disallows split counts only files where both the Googlebot and * groups have at least one disallow. Tiers use Chrome UX Report rank.
Using this data?
## Cite this study
Link back to the study when you cite it.
APA
Matt Suffoletto. (2026). AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%. PageSpeed Matters. https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt
Plain link
AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3% - PageSpeed Matters - https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt
## Related studies
- Technical SEO Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File: https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics
- Technical SEO Robots.txt Mistakes: 21.7% Use Rules Google Ignores: https://www.pagespeedmatters.com/resources/data-studies/robots-txt-mistakes
- AI Search Sites Cited in AI Overviews Pass Core Web Vitals 56.8% of the Time vs 40% Web-Wide: https://www.pagespeedmatters.com/resources/data-studies/ai-overview-citation-speed
- Technical SEO XML Sitemap Errors: 1 in 4 Sampled Sites List URLs Google Shouldn't Index: https://www.pagespeedmatters.com/resources/data-studies/xml-sitemap-errors
## Ready to Get Started?
Get your site optimized in as little as 3 days. From $500.
See Plans & Pricing: https://www.pagespeedmatters.com/pricing
## Structured data
```json
[
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%",
"datePublished": "2026-10-19",
"dateModified": "2026-10-19",
"author": {
"@type": "Person",
"name": "Matt Suffoletto",
"jobTitle": "Performance Optimization Expert",
"url": "https://www.pagespeedmatters.com/resources/author/matt-suffoletto"
}
},
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://www.pagespeedmatters.com/"
},
{
"@type": "ListItem",
"position": 2,
"name": "Resources",
"item": "https://www.pagespeedmatters.com/resources"
},
{
"@type": "ListItem",
"position": 3,
"name": "Data Studies",
"item": "https://www.pagespeedmatters.com/resources/data-studies"
},
{
"@type": "ListItem",
"position": 4,
"name": "AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%"
}
]
},
{
"@context": "https://schema.org",
"@type": "Dataset",
"name": "AI Crawlers in Robots.txt: GPTBot 11.8%, Googlebot 5.3%",
"description": "Of 13,358,406 robots.txt files returning a 200, 11.8% name GPTBot, 10.9% ClaudeBot and 5.3% Googlebot. Among Top 10K sites, 23.8% name GPTBot.",
"url": "https://www.pagespeedmatters.com/resources/data-studies/ai-crawlers-in-robots-txt",
"creator": {
"@type": "Organization",
"name": "PageSpeed Matters",
"url": "https://www.pagespeedmatters.com"
},
"datePublished": "2026-10-19",
"dateModified": "2026-10-19",
"license": "https://creativecommons.org/licenses/by/4.0/",
"isAccessibleForFree": true,
"creativeWorkStatus": "Published",
"variableMeasured": "Core Web Vitals pass rate and web performance metrics",
"measurementTechnique": "HTTP Archive crawl, September 2026: robots.txt files for all origins"
},
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "How many websites mention GPTBot in robots.txt?",
"acceptedAnswer": {
"@type": "Answer",
"text": "11.8% of 13,358,406 robots.txt files returning a 200 name GPTBot in a user-agent group, rising to 23.8% among Top 10K sites. That counts files with a rule for GPTBot, not only those that block it."
}
},
{
"@type": "Question",
"name": "Do more sites name AI crawlers than Googlebot?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. 11.8% of files name GPTBot and 10.9% name ClaudeBot, against 5.3% that name Googlebot. Google-Extended, Google's AI training token, is named in 9.8%."
}
},
{
"@type": "Question",
"name": "Does adding a Googlebot group to robots.txt change the * rules for Google?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes, Googlebot follows its own group instead of the * group. Of 676,853 files with both, 26.4% give Googlebot no disallows while the * group has some."
}
},
{
"@type": "Question",
"name": "How do I check which crawlers my robots.txt addresses?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Open yoursite.com/robots.txt and read each user-agent line. Every crawler named there follows only its own group, and every other crawler follows the * group."
}
}
]
}
]
```