Almost every panic about ecommerce crawl budget starts from the wrong premise: that Google has capped how many of your pages it will crawl, and you need to beg for more. That’s not the real problem. Googlebot will happily crawl a huge, fast, well-structured store all day. The problem on large catalogs is almost never a shortage of crawling — it’s that a finite crawl allowance gets poured into millions of near-worthless URLs (filter combinations, sort variants, session parameters) while your actual product and category pages get visited once a month. Crawl budget isn’t something you top up. It’s something you stop leaking.
Crawl Budget Is Rarely the Problem — Crawl Waste Is
Reframe the whole issue and the fix becomes obvious. When a new product takes three weeks to get indexed, or a price change doesn’t show in search for a fortnight, the instinct is “Google won’t crawl my site enough.” Look at the server logs and you usually find the opposite: Googlebot is crawling tens of thousands of URLs a day — it’s just spending 90% of that effort on ?color=red&sort=price-asc&page=7 permutations that should never have been crawlable in the first place. The money pages starve not because the budget is small, but because the budget is being spent on garbage. Fix the leak and the same crawl volume suddenly covers what matters.
What Ecommerce Crawl Budget Actually Means
Google defines crawl budget as the product of two things: crawl capacity (how many simultaneous connections your server can handle without slowing down, and how fast it responds) and crawl demand (how much Google wants your URLs, based on popularity and how often they change). A fast server raises the ceiling; genuinely useful, frequently updated pages raise the demand. For a large store the practical takeaway is blunt: you influence crawl budget far more by improving server response time and by not creating junk URLs than by fiddling with any single directive.
Do You Even Have a Crawl Budget Problem?
Most stores don’t, and over-engineering this wastes weeks. Google has said crawl budget is a real concern mainly for sites with more than roughly a million unique URLs, or medium sites (10,000+ URLs) that change often. Here’s a simple decision rule:
- Count your real pages (products + categories + content) versus your crawlable URLs. If crawlable URLs are 10× or 50× your real page count, faceted navigation is manufacturing crawl waste.
- Check indexing latency: if new products reliably index within a few days, you don’t have a crawl problem worth solving yet.
- Check “Crawled – currently not indexed” and “Discovered – currently not indexed” in Search Console. Large numbers here on a big catalog are the classic signature of a large store crawl budget leak.
If you’re a 300-product boutique, stop reading and go write better product descriptions. This article is for catalogs where the URL count has quietly exploded past what your content can justify.
Where Large Catalogs Bleed Crawl Budget
The waste is almost always the same short list of culprits, and naming them precisely is half the fix:
- Faceted-navigation URLs — every filter and sort combination generates a unique, crawlable URL. Ten filters with a few values each multiply into millions of permutations.
- Internal search result pages —
?q=URLs that generate infinite thin pages Google should never index. - Session IDs and tracking parameters appended to otherwise-identical pages, each looking like a new URL.
- Near-duplicate variant pages — the same product split across colour or size URLs with 95% identical content.
- Redirect chains and soft 404s — discontinued products bouncing through three hops, or “no results” pages returning HTTP 200.
- Endless pagination on huge category listings where deep pages hold little unique value.
The Faceted-Navigation Trap
Faceted navigation is where most ecommerce crawling problems are born, so it deserves its own section. Filters are great for shoppers and toxic for crawlers when every combination is a linked, indexable URL. A category with filters for size, colour, brand, price band and an on-page sort control can generate combinatorially — tens of thousands of URLs from a single category that only ever needed one. Googlebot follows those internal links, crawls the permutations, and burns your budget discovering pages that are duplicates or empty result sets.
The durable pattern most large retailers converge on: decide which facets have genuine search demand (e.g. “nike running shoes” — brand + category is a real query worth a static, indexable landing page) and which are pure UX (sort order, arbitrary price sliders). Make the valuable facets clean, static, indexable URLs you actively want ranked; keep the rest out of the crawl path entirely.
robots.txt vs noindex vs Canonical: Which Actually Saves Crawls
This is the mechanism nearly every guide gets muddled, and getting it wrong wastes months. The three tools do different jobs:
- robots.txt
Disallowis the only one that actually prevents crawling — and therefore the only one that truly saves crawl budget. Block the facet parameters and internal-search paths here and Googlebot stops fetching them. noindexkeeps a page out of the index but does not save crawl budget — Google has to crawl the page to see the noindex tag in the first place. Using noindex alone to “fix” faceted URLs still lets Googlebot burn budget on every one.- Canonical tags consolidate ranking signals between duplicates but, again, don’t stop crawling — Google still fetches the URL to read the canonical.
The nuance that trips people: a robots.txt-disallowed URL can still appear in the index (as a bare link with no snippet) if enough internal or external links point to it, because Google never crawls it to see a noindex. So the clean sequence is — stop linking to junk facets internally, disallow the parameter patterns in robots.txt, and reserve noindex for the handful of thin pages you must keep crawlable but out of the index. Don’t combine noindex and robots.txt on the same URL, or Google can’t crawl it to honour the noindex.
Out-of-Stock, Discontinued, and Redirect Chains
Product lifecycle handling quietly drains crawl efficiency on any large store. The honest rules: a product that’s temporarily out of stock should stay live at HTTP 200 (removing it throws away accumulated authority and links). A product gone for good should return 404 or 410 — a 410 tells Google to drop it faster. Never redirect a dead product to an unrelated homepage or category; that’s a soft 404 Google increasingly treats as a low-quality signal, and it wastes a crawl. Audit for redirect chains too — every extra hop is an extra crawl Google spends before reaching content, and at catalog scale those hops add up to real wasted budget.
Diagnose With Log Files, Not Guesswork
You cannot fix crawl waste you can’t see, and Search Console’s Crawl Stats report is only a summary. The real diagnostic is server log analysis: filter for verified Googlebot and look at exactly which URL patterns it fetches, how often, and how much of that hits parameter junk versus product pages. Nine times out of ten the logs reveal that a huge share of crawls land on facet or search URLs — which instantly tells you what to disallow. This is also the fastest way to prove a fix worked: watch the share of Googlebot hits on your product URLs climb after you close the leaks.
A real-crawler site audit accelerates the discovery step. SEO Rocket’s site audit crawls your store the way a bot does and surfaces the exact ecommerce failure modes — duplicate variant pages, thin product descriptions, redirect chains, broken links, and parameter-bloated URL patterns — so you can see where crawl budget is leaking before you ever open a log file. It’s an SEO diagnostic layer, not a store platform, but it points you straight at the URLs eating your ecommerce crawl budget.
Make Every Crawl Count
Once the leaks are closed, the goal flips to crawl efficiency — helping Googlebot reach and re-crawl what matters:
- Clean XML sitemaps listing only canonical, indexable URLs, with accurate
lastmoddates so Google re-crawls pages that genuinely changed (a price or stock update) and skips the rest. - Faster server responses. Crawl capacity scales with speed — a store that answers in 200ms gets crawled harder than one that stalls at two seconds.
- Shallow internal linking so key products sit a few clicks from the homepage; deeply buried pages get less crawl demand.
- Kill thin content. Pages with a bare title and manufacturer boilerplate carry low crawl demand and drag on quality. This is where SEO Rocket’s validation-gated AI writer earns its place — generating genuinely unique product descriptions and buying guides at scale, with length and quality gates, so a 40,000-SKU catalog isn’t shipping duplicate filler that Google deprioritises.
A Worked Example: 40,000 Products, 6 Million Crawlable URLs
Picture a mid-size retailer — illustrative numbers only. The catalog holds around 40,000 products across 800 categories, so the “real” page count is roughly 45,000. But each category exposes six filters plus a sort control, and internal search is crawlable. The logs show Googlebot fetching well over 100,000 URLs a day, the vast majority of them facet permutations, while individual products get re-crawled maybe once a month and new arrivals sit in “Discovered – currently not indexed” for weeks.
The fix isn’t asking Google for more crawling. It’s: disallow the filter and search parameters in robots.txt, stop linking to non-indexable facets in the navigation, promote the handful of high-demand facet pages to clean static URLs, 410 the discontinued products, and submit a lastmod-accurate sitemap of only the 45,000 real URLs. Same crawl volume — now aimed at pages that earn revenue. Indexing latency for new products typically collapses from weeks to days once the junk stops competing for the same finite attention. That’s the entire game of crawl efficiency for ecommerce: not more crawling, better-aimed crawling.
Frequently Asked Questions
Does noindex save crawl budget on ecommerce sites?
No. Google has to crawl a URL to see its noindex tag, so noindex keeps a page out of the index but still spends a crawl on it every time. To actually save crawl budget on faceted or search URLs, block the pattern in robots.txt so Googlebot never fetches it — and stop linking to it internally so it isn’t discovered.
How many pages before crawl budget matters for a store?
Google indicates crawl budget becomes a real concern above roughly a million unique URLs, or for medium sites (10,000+ URLs) that change frequently. The sharper test is the ratio: if your crawlable URLs vastly outnumber your real products, or new products take weeks to index, you have a waste problem worth fixing regardless of raw page count.
Should out-of-stock products be removed to save crawl budget?
Not if the item is coming back — keep it live at HTTP 200 to preserve its links and authority. For products gone permanently, return 410 (or 404) so Google drops them. Redirecting dead products to the homepage creates soft 404s that waste crawls and signal low quality, so avoid it.
