Most sitemap advice assumes a tidy picture: one sitemap.xml at the root, every URL stamped with a fresh date, and a line in robots.txt pointing to it. Our xml sitemap study of 1,000 of the most popular websites found something messier. On 8 November 2026 we crawled the top of the Tranco ranking and checked how each site publishes its sitemap. 644 of the 1,000 declare a sitemap in robots.txt. 413 of the 657 sitemaps we could parse are sitemap indexes, not single files. And only 376 of those 657 (57.2%) use <lastmod> at all, even though it is the one optional tag both Google and Bing say they use.
This is original data from a single crawl, and we explain below exactly what we fetched and what we could not see. It belongs to a four-part series on the same 1,000 sites, alongside our AI crawler blocking study, our llms.txt adoption study and our homepage schema study.
The findings at a glance
Every count below comes from one crawl of 1,000 websites. The chart shows each result as a share of all 1,000 sites. Some figures in the text use a smaller base, such as the 657 sitemaps we could parse, and we say so each time.
- 644 of 1,000 sites (64.4%) declare at least one sitemap in robots.txt. That is 71.2% of the 905 sites that returned a robots.txt file.
- 256 sites (25.6%) declare more than one sitemap in robots.txt. One robots.txt file lists 208.
- 657 sites (65.7%) had a first sitemap we could fetch and parse as XML.
- Of those 657, 413 (62.9%) are sitemap indexes and 244 (37.1%) are single
urlsetfiles. The median index lists 18 child sitemaps. - 376 of 657 (57.2%) parsed sitemaps contain
<lastmod>. - Only 44 of 657 (6.7%) declare the image sitemap namespace, and 41 (6.2%) include hreflang alternates.
- 343 sites (34.3%) had no sitemap we could parse, either at the first URL declared in robots.txt or, where none was declared, at
/sitemap.xml.
What we measured, briefly
For each site we fetched /robots.txt, counted its Sitemap: lines and then fetched the first sitemap listed. If robots.txt listed none, we tried /sitemap.xml. We recorded whether the file was a sitemap index or a urlset, counted its <loc> entries, and checked for <lastmod>, the image namespace and hreflang links. We did not open child sitemaps or check whether any dates were accurate. The full method and its limits are in the Methodology section near the end.
Sitemaps in robots.txt: the default at the top of the web
The robots.txt Sitemap: line is the simplest way to make a sitemap discoverable to every crawler at once, not just the ones you have submitted it to. The sitemaps.org protocol says the directive is independent of the user-agent line, so it can go anywhere in the file. A typical declaration looks like this:
Sitemap: https://www.example.com/sitemap_index.xml
644 of the 1,000 sites (64.4%) do this. Measured against the 905 sites that returned a robots.txt at all, the rate is 71.2%. Put the other way, more than a quarter of the robots.txt files we read point no crawler to any sitemap. Search Console submission covers Google and Bing Webmaster Tools covers Bing, but robots.txt is a place any crawler can find it, including newer AI search crawlers and SEO tools. If you already maintain a robots.txt, one extra line is the cheapest discovery win available. Our robots.txt guide covers placement and syntax. The robots.txt tester confirms the file parses cleanly.
One sitemap is rarely enough: 256 sites declare several
256 sites (25.6% of 1,000, or 39.8% of the 644 that declare any) list more than one sitemap in robots.txt. The most we saw in a single file was 208 separate Sitemap: lines.
That is allowed. The sitemaps.org protocol says you can specify more than one sitemap per robots.txt file. Google’s sitemap documentation goes further and says there is no limit to the number of sitemaps you can include there. Big sites often split by section, language, content type or business unit, and each team may own its own sitemap. The trade-off is upkeep. A long list of hand-maintained lines tends to break when a section is retired or a URL pattern changes. A single sitemap index that points to the children usually does the same job with one line to maintain.
Sitemap index vs single urlset: indexes win at scale
Of the 657 sites where we could parse the first sitemap, 413 (62.9%) served a sitemap index, a file that lists other sitemaps rather than pages. The remaining 244 (37.1%) served a single urlset of page URLs. The median index pointed to 18 child sitemaps.
The protocol explains why indexes dominate on large sites. A single sitemap is capped at 50,000 URLs and 50 MB uncompressed, and Google states the same limits. A site with millions of product, profile or article URLs has no choice but to split its sitemap, and an index ties the pieces together. Indexes also help with diagnosis. When child sitemaps are split by template or section, Search Console reports indexing per file, so you can see which section is underperforming rather than one blended number. Our XML sitemap guide covers how to structure the split.
If you are starting from scratch, the pattern at the top of the web is clear: an index at a stable URL, declared in robots.txt, with children grouped in a way that matches how you report on the site. For the basics of what goes inside each file, see what is an XML sitemap.
lastmod usage: the tag that matters, used by 57.2%
Of the 657 parsed sitemaps, 376 (57.2%) contain at least one <lastmod> tag. The other 42.8% of the files we parsed give crawlers no date signal at all.
That gap matters because <lastmod> is the one optional sitemap tag the major engines say they use. Google’s documentation says it uses <lastmod> if the value is “consistently and verifiably” accurate, and that it ignores <priority> and <changefreq>. Bing’s July 2025 webmaster post describes lastmod as a key signal that helps Bing prioritize URLs for recrawling and reindexing. Bing also warns against stamping every URL with the time the sitemap was generated.
Two caveats about our number. First, we checked whether the tag is present, not whether it is honest. A sitemap that sets every date to today counts as “uses lastmod” here, but it gives crawlers a signal they will learn to ignore. Second, for the 413 indexes we checked the index file only. In an index, <lastmod> sits on the child-sitemap entries, so a site could date its pages inside child files and still count as “no lastmod” in our data. The practical takeaway still holds: date the URLs that really changed, and only when the main content changed.
Image, hreflang and gzip: the rarer extensions
Sitemap extensions are uncommon in our sample. 44 of 657 parsed sitemaps (6.7%) declare the image sitemap namespace, and 41 (6.2%) include hreflang alternate links. Both numbers are floors, not totals. Most first sitemaps were indexes, and these extensions usually live in the child files we did not open.
Hreflang in sitemaps is one of three valid ways to declare language alternates. Large multilingual sites often choose it because it keeps hundreds of alternate tags out of the page head. Our hreflang sitemap guide walks through the markup.
Compression is rarer still. For 29 of the 1,000 sites (2.9%), the first sitemap URL is a .gz file. The protocol allows gzip as long as the file stays within 50 MB uncompressed. It saves bandwidth on very large sitemaps, but it also makes the file harder to inspect by hand, which is one reason tooling matters at that scale.
The 343 sites with no sitemap we could parse
343 sites (34.3%) returned no parseable sitemap, either at the first URL declared in robots.txt or, where none was declared, at /sitemap.xml. Do not read that as “a third of top sites have no sitemap.” The Tranco list is weighted toward platforms and infrastructure brands, and many of these sites are apps and login-first products with little public content to list. A logged-in dashboard has no reason to publish a sitemap.
Some of the 343 are measurement gaps rather than missing sitemaps. Our crawler only tried the first declared URL and did not fall back to /sitemap.xml if that URL failed. Some servers refuse unfamiliar crawlers or return errors to them. Gzipped files served without a compression header could not be parsed. And a sitemap that lives at a custom path, without a robots.txt line, is invisible to a check like ours. A crawler that does not know your submission history faces the same problem. That is another argument for the robots.txt line.
What this XML sitemap study means for your site
The top of the web is not a model of perfect sitemap hygiene, but its habits point to a sensible baseline:
- Declare your sitemap in robots.txt. 64.4% of top sites do it, and it costs one line.
- Use a sitemap index once you outgrow one file, and split the children by section or template so indexing reports tell you something useful.
- Use
<lastmod>, and keep it accurate. 42.8% of the sitemaps we parsed skip it. A fake date is worse than none, because crawlers learn to distrust it. - Drop
<priority>and<changefreq>expectations. Google and Bing both say they ignore them. - Validate after every deploy. Run the file through our free XML sitemap validator, list what is really inside with the sitemap URL extractor, then submit it to Google.
Sitemaps are also a fast way to understand someone else’s site. SEO Rocket’s site-structure feature maps any website from its sitemaps, so you can see how a competitor groups its sections and how big each one is without crawling every page. When you audit your own site, SEO Rocket’s site audit runs a real crawler and flags broken, redirecting and noindexed URLs, which are exactly the URLs that should not sit in a sitemap. Keeping them out matters for crawl budget on large sites.
Methodology and limitations
Sample. We started from the Tranco top-1M list (list ID Q2VV4), a research ranking that averages several popularity lists. On 8 November 2026 we crawled ranks 1 to 2,300 and kept the first 1,000 unique websites. These were domains whose homepage returned HTML, de-duplicated by final origin after redirects, so country domains that redirect to one global site count once. Many top-ranked domains are CDNs, APIs or ad hosts with no website, which is why 2,300 ranks produced 1,000 sites. The last site included is rank 2,293.
Fetching. For each site we fetched /robots.txt, which 905 of the 1,000 sites returned, and counted its Sitemap: lines. We then fetched the first declared sitemap, or /sitemap.xml if none was declared. Each URL was fetched once by a self-identifying research crawler (SEORocketStudy/1.0) with a 10-second timeout, reading up to 5 MB of each sitemap. A file counted as parsed if it contained a sitemapindex or urlset element. “Uses lastmod”, “image” and “hreflang” are presence checks on that first file only.
Limitations. This is a single snapshot. We read only the first sitemap per site and never opened child sitemaps, so lastmod, image and hreflang figures understate what deeper sitemaps contain. We checked whether lastmod is present, not whether it is accurate. Some sites serve bots different content or block unknown crawlers, and gzipped files without a compression header could not be read. Both push the “no sitemap found” count up. Tranco is a popularity proxy weighted toward big platforms and infrastructure brands, not a sample of “the web”, and small-business sites behave differently. Unless another base is stated, percentages are of 1,000 sites. Contact us for the dataset.
FAQ
How many top websites declare a sitemap in robots.txt?
In our crawl of 1,000 top sites, 644 (64.4%) declare at least one sitemap in robots.txt. Among the 905 sites that returned a robots.txt file, that is 71.2%.
Do most large sites use a sitemap index?
Yes. Of the 657 sites whose first sitemap we could parse, 413 (62.9%) served a sitemap index rather than a single urlset. The median index listed 18 child sitemaps.
Is lastmod still worth adding to a sitemap?
Yes, if it is accurate. Google says it uses lastmod when the value is consistently and verifiably accurate, and Bing calls it a key recrawl signal. Only 376 of 657 parsed sitemaps (57.2%) in our sample include it.
