---
title: "X-Robots-Tag Noindex: 30.6% of Noindexes Use a Header"
url: https://www.pagespeedmatters.com/resources/data-studies/x-robots-tag-noindex
description: "12,177 top 1M homepages are noindexed and 30.6% do it with an X-Robots-Tag header. Shopify uses the header 78.9% of the time, WordPress 23.9%."
lang: en
---
Data Study ยท Technical SEO
# X-Robots-Tag Noindex: 3 in 10 Noindexed Homepages Hide It in a Header
12,177 homepages in the top 1M sites by Chrome traffic tell Google not to index them, and 30.6% of those (3,722) do it with an X-Robots-Tag HTTP header instead of a meta tag.
- 782,850 top 1M homepages
- HTTP Archive, Sept 2026
Matt Suffoletto: https://www.pagespeedmatters.com/resources/author/matt-suffoletto
Founder, PageSpeed Matters Published Oct. 19, 2026 6 min read
## Key Findings
1. **12,177** homepages in the top 1M sites by Chrome traffic are noindexed, and 3,722 of them (**30.6%**) do it with an X-Robots-Tag header rather than a meta tag in the page. Copy stat
2. 1,376 of the 3,722 header-noindexed homepages (**37.0%**) still have real Chrome user traffic in CrUX, against **46.2%** of homepages noindexed by meta tag (3,906 of 8,455). Copy stat
3. Noindexed Shopify homepages almost always use the header: 138 of 175 (**78.9%**), against **23.9%** of 376 noindexed WordPress homepages and 30.1% of 11,606 noindexed homepages on other or custom stacks. Copy stat
4. Noindexed homepages are rare on the big platforms: **0.37%** of 47,127 Shopify homepages and 0.33% of 113,593 WordPress homepages are noindexed, against **1.88%** of 618,129 other or custom homepages (1.56% overall). Copy stat
5. **1,844** of the 12,177 noindexed homepages (**15.1%**) are indexable in the raw HTML and only become noindex after JavaScript runs. Copy stat
6. Header noindexes are more common further down the popularity curve: **19.7%** of 66 noindexed Top 10K homepages use the header, against 27.7% of 1,103 in the Top 100K and **30.9%** of 11,008 in the Top 1M tier. Copy stat
Share of noindexed homepages using the X-Robots-Tag header, by tier
Top 10K (66) 19.7%
Top 100K (1,103) 27.7%
Top 1M (11,008) 30.9%
Source: PageSpeed Matters. HTTP Archive crawl, Sept 2026, mobile, top 1M sites by Chrome traffic.
Tiers are separate rank bands (Top 100K means ranks 10K to 100K). The Top 10K group is small and includes the 3 noindexed Top 1K homepages.
In this study
## How noindexed top 1M homepages block indexing, by platform
| Platform | Noindexed homepages | Share noindexed | Via X-Robots-Tag header | Homepages |
| --- | --- | --- | --- | --- |
| Shopify | 175 | 0.37% | 78.9% | 47,127 |
| WordPress | 376 | 0.33% | 23.9% | 113,593 |
| Other or custom | 11,606 | 1.88% | 30.1% | 618,129 |
| All homepages | 12,177 | 1.56% | 30.6% | 782,850 |
Source: PageSpeed Matters, Sept 2026. Webflow, Wix and Squarespace had too few noindexed homepages to report and count only in the All homepages row.
## Summary
There are two ways to tell Google not to index a page. Most people know the robots meta tag in the page's HTML. The other is the X-Robots-Tag HTTP header, which the server sends before any HTML. It is easy to miss, because it does not show up when you view the page source.
We looked at homepages in the top 1M sites by Chrome traffic. 12,177 of them tell Google not to index the homepage, and 30.6% of those (3,722), about 3 in 10, do it with an X-Robots-Tag header instead of a meta tag. On Shopify, the header is the usual route.
## What we measured
We used the September 2026 HTTP Archive mobile crawl of the top 1M sites by Chrome traffic. For each of 782,850 homepages, we read the rendered index status for Googlebot and where a noindex came from: a meta tag in the page or an X-Robots-Tag response header. We also compared raw and rendered HTML to see whether a noindex only appears after JavaScript runs, and checked Chrome UX Report (CrUX) field data to see which still get real Chrome visitors.
Some of these noindexes are intentional. Login portals, web apps and staging hosts often should not be indexed, so not every case below is a mistake. Webflow, Wix and Squarespace had too few noindexed homepages to report on their own, so they appear only in the overall totals. The Top 10K tier includes the 3 noindexed Top 1K homepages.
## Three in ten noindexed homepages use a header
Of the 12,177 noindexed homepages, 3,722 (30.6%) send the instruction in an X-Robots-Tag header rather than a meta tag in the page. Google treats both the same way, but the header is harder to spot. A developer or SEO looking at the page source will not see it, and many browser SEO extensions only check the HTML.
Many of these homepages still get visitors. 1,376 of the 3,722 header-noindexed homepages (37.0%) have real Chrome user traffic in CrUX. For homepages noindexed by meta tag the share is higher: 46.2% (3,906 of 8,455). Some of these are deliberate, such as account portals with plenty of logged-in users. But a homepage with real traffic and a noindex is worth a second look, because it may be a live site that is quietly missing from search.
30.6%
Share of the 12,177 noindexed top 1M homepages that block indexing with an X-Robots-Tag header instead of a meta tag.
## Shopify homepages almost always use the header
How a homepage gets noindexed depends heavily on the platform. Noindexed Shopify homepages almost always use the header: 138 of 175 (78.9%). On WordPress the header is the exception, at 23.9% of 376 noindexed homepages, and on other or custom stacks it is 30.1% of 11,606.
The Shopify figure likely reflects the noindex being set at the server level, by the platform or an app, rather than in the theme template. That matters for store owners, because removing a meta tag from a theme will not help if the header is still being sent.
Noindexed homepages are rare on the big platforms in the first place. Only 0.37% of 47,127 Shopify homepages and 0.33% of 113,593 WordPress homepages are noindexed, against 1.88% of 618,129 homepages on other or custom stacks, and 1.56% overall.
## Some homepages only become noindex after JavaScript
1,844 of the 12,177 noindexed homepages (15.1%) are indexable in the raw HTML the server sends and only become noindex after JavaScript runs. In those cases the instruction a crawler sees depends on whether it renders the page.
That is a fragile setup. Google's JavaScript SEO guidance warns that changing the robots meta tag with JavaScript may not work as expected, and crawlers that do not render JavaScript will see an indexable page. If a homepage should not be indexed, the safer place for that instruction is the raw HTML or the response header.
## Header noindexes are more common on less popular sites
The header is used more often as sites get less popular. 19.7% of the 66 noindexed Top 10K homepages use the X-Robots-Tag header. That rises to 27.7% of 1,103 noindexed homepages in the Top 100K tier (sites ranked 10K to 100K) and 30.9% of 11,008 in the Top 1M tier (ranked 100K to 1M). Each tier is a separate rank band, so the three groups add up to all 12,177 noindexed homepages.
The Top 10K group is small, so treat that end of the chart with care. The broader point holds at every tier: a meaningful share of noindexes never appear in the HTML, so an audit that only reads page source will miss them.
## What this means for your site
If a page is not showing up in Google and the source looks clean, check the response headers. Open your browser's network panel, select the page request and look for X-Robots-Tag in the response headers, or run a header checker. Search Console's URL Inspection tool will also tell you whether indexing is blocked and whether the block came from a header or a meta tag.
Then trace where the header comes from. Common sources are server or CDN rules, hosting settings for staging environments that were never switched off, security plugins and store apps. On a store, check installed apps before editing the theme.
Finally, keep noindex out of JavaScript. If a page should be indexed, make sure no script adds a noindex after load. If it should not be, put the rule in the raw HTML or the header so every crawler sees the same thing.
## Sources
- HTTP Archive crawl, September 2026, mobile, top 1M sites by Chrome traffic, with Chrome UX Report (CrUX) field data
- Google Search Central, robots meta tag and X-Robots-Tag specifications: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
## Embed this research
Paste this on your site to embed the chart. It links back to the source automatically.
Source: X-Robots-Tag Noindex: 3 in 10 Noindexed Homepages Hide It in a Header - PageSpeed Matters
## Frequently asked questions
### What is the X-Robots-Tag header?
It is an HTTP response header that carries the same indexing rules as the robots meta tag, such as noindex. Because it is sent with the response rather than in the HTML, it does not appear in the page source.
### How common is X-Robots-Tag noindex?
Of 12,177 noindexed homepages in the top 1M sites by Chrome traffic, 30.6% (3,722) use the X-Robots-Tag header instead of a meta tag. On Shopify the share is 78.9% of noindexed homepages, against 23.9% on WordPress.
### Can JavaScript add a noindex to a page?
Yes, and it happens: 1,844 of the 12,177 noindexed homepages (15.1%) are indexable in the raw HTML and only become noindex after JavaScript runs. Whether a crawler sees that noindex depends on whether it renders the page.
### How do I check if a page has an X-Robots-Tag noindex?
Look at the response headers in your browser's network panel or with a header checker, and search for X-Robots-Tag. Search Console's URL Inspection tool also reports whether indexing is blocked and where the block comes from.
A stray noindex header is easy to miss and easy to remove once you know where the server, CDN or app is adding it. Need help with this? See our technical SEO services (https://www.pagespeedmatters.com/services/technical-seo).
Share
How we measured
## Methodology & Source
Data source
HTTP Archive crawl of the top 1M sites by Chrome traffic (mobile, Sept 2026) with Chrome UX Report field data
Collection date
September 2026
Sample
782,850 top 1M homepages
Release
HTTP Archive, Sept 2026
Rendered Googlebot index status and its source (meta tag or X-Robots-Tag header) for 782,850 homepages. Some noindexes are intentional, such as login portals and apps, so not every case is a mistake. The Top 10K figure includes the 3 noindexed Top 1K homepages. Webflow, Wix and Squarespace had too few noindexed homepages to report and appear only in the overall totals.
Using this data?
## Cite this study
Link back to the study when you cite it.
APA
Matt Suffoletto. (2026). X-Robots-Tag Noindex: 3 in 10 Noindexed Homepages Hide It in a Header. PageSpeed Matters. https://www.pagespeedmatters.com/resources/data-studies/x-robots-tag-noindex
Plain link
X-Robots-Tag Noindex: 3 in 10 Noindexed Homepages Hide It in a Header - PageSpeed Matters - https://www.pagespeedmatters.com/resources/data-studies/x-robots-tag-noindex
## Related studies
- Website Migration 43% of Homepages Hidden From Google Still Get Real Traffic: https://www.pagespeedmatters.com/resources/data-studies/accidental-noindex
- Technical SEO Robots.txt Statistics 2026: 1 in 6 Sites Lack a Working File: https://www.pagespeedmatters.com/resources/data-studies/robots-txt-statistics
- Technical SEO JavaScript Title Rewrites: 12.6% of Top 1M Homepages: https://www.pagespeedmatters.com/resources/data-studies/javascript-title-rewrites
- Technical SEO Canonical Tag Usage 2026: 36.1% of Inner Pages Have None: https://www.pagespeedmatters.com/resources/data-studies/canonical-tag-usage
## Ready to Get Started?
Get your site optimized in as little as 3 days. From $500.
See Plans & Pricing: https://www.pagespeedmatters.com/pricing
## Structured data
```json
[
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "X-Robots-Tag Noindex: 3 in 10 Noindexed Homepages Hide It in a Header",
"datePublished": "2026-10-19",
"dateModified": "2026-10-19",
"author": {
"@type": "Person",
"name": "Matt Suffoletto",
"jobTitle": "Performance Optimization Expert",
"url": "https://www.pagespeedmatters.com/resources/author/matt-suffoletto"
}
},
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://www.pagespeedmatters.com/"
},
{
"@type": "ListItem",
"position": 2,
"name": "Resources",
"item": "https://www.pagespeedmatters.com/resources"
},
{
"@type": "ListItem",
"position": 3,
"name": "Data Studies",
"item": "https://www.pagespeedmatters.com/resources/data-studies"
},
{
"@type": "ListItem",
"position": 4,
"name": "X-Robots-Tag Noindex: 3 in 10 Noindexed Homepages Hide It in a Header"
}
]
},
{
"@context": "https://schema.org",
"@type": "Dataset",
"name": "X-Robots-Tag Noindex: 3 in 10 Noindexed Homepages Hide It in a Header",
"description": "12,177 top 1M homepages are noindexed and 30.6% do it with an X-Robots-Tag header. Shopify uses the header 78.9% of the time, WordPress 23.9%.",
"url": "https://www.pagespeedmatters.com/resources/data-studies/x-robots-tag-noindex",
"creator": {
"@type": "Organization",
"name": "PageSpeed Matters",
"url": "https://www.pagespeedmatters.com"
},
"datePublished": "2026-10-19",
"dateModified": "2026-10-19",
"license": "https://creativecommons.org/licenses/by/4.0/",
"isAccessibleForFree": true,
"creativeWorkStatus": "Published",
"variableMeasured": "Core Web Vitals pass rate and web performance metrics",
"measurementTechnique": "HTTP Archive crawl of the top 1M sites by Chrome traffic (mobile, Sept 2026) with Chrome UX Report field data"
},
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is the X-Robots-Tag header?",
"acceptedAnswer": {
"@type": "Answer",
"text": "It is an HTTP response header that carries the same indexing rules as the robots meta tag, such as noindex. Because it is sent with the response rather than in the HTML, it does not appear in the page source."
}
},
{
"@type": "Question",
"name": "How common is X-Robots-Tag noindex?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Of 12,177 noindexed homepages in the top 1M sites by Chrome traffic, 30.6% (3,722) use the X-Robots-Tag header instead of a meta tag. On Shopify the share is 78.9% of noindexed homepages, against 23.9% on WordPress."
}
},
{
"@type": "Question",
"name": "Can JavaScript add a noindex to a page?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes, and it happens: 1,844 of the 12,177 noindexed homepages (15.1%) are indexable in the raw HTML and only become noindex after JavaScript runs. Whether a crawler sees that noindex depends on whether it renders the page."
}
},
{
"@type": "Question",
"name": "How do I check if a page has an X-Robots-Tag noindex?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Look at the response headers in your browser's network panel or with a header checker, and search for X-Robots-Tag. Search Console's URL Inspection tool also reports whether indexing is blocked and where the block comes from."
}
}
]
}
]
```