Your crawl budget is the number of URLs Googlebot is willing and able to fetch from your site in a given period. It is not a number you can look up, buy, or negotiate. It emerges from two things Google measures continuously: how much load your server can take without slowing down, and how much Google thinks your content is worth revisiting.
Before you spend a week on this, a blunt filter. If your site has fewer than a few thousand URLs and pages get indexed within a day or two of publishing, crawl budget seo is not your problem. Google’s own guidance is that most sites never need to think about it. Chase your rankings instead. The advice below is for sites where the numbers actually bite: large ecommerce catalogs, classifieds, marketplaces, news archives, and anything generating URLs from filters.
The two halves: crawl capacity and crawl demand
Google splits what is crawl budget into two components, and confusing them leads to the wrong fix.
Crawl capacity limit is a server health ceiling. Googlebot ramps up request rate until response times degrade or it starts hitting 5xx errors, then backs off. A site returning 180ms responses gets crawled far harder than one averaging 1.8 seconds. This half is an infrastructure problem, and it is the one most people underestimate.
Crawl demand is interest. Popular URLs with links pointing at them get refetched often. Pages that have not changed in three years and nobody links to get visited rarely. Stale inventory, duplicate clusters, and abandoned sections all sit at the bottom of the queue — which is fine, until they start consuming requests that should have gone to new products.
You raise capacity with faster infrastructure. You raise demand with better content and more links. Everything else in this article is about waste elimination: stopping the crawler from spending its allowance on URLs that will never rank.
Where crawl budget actually leaks
In practice, the same handful of patterns account for most waste on large sites:
- Faceted navigation. Five filters with ten values each can produce more URL combinations than you have products. This is the single biggest source of crawl budget optimization for ecommerce work.
- Session IDs and tracking parameters in URLs. Every campaign tag creates a fresh address. Remember that the Search Console URL Parameters tool was retired in 2022 — there is no dashboard fix anymore.
- Internal redirect chains. Each hop is a separate fetch. A three-hop chain triples the cost of reaching one page.
- Soft 404s. Empty search results and out-of-stock pages returning 200 instead of 404 or 410 stay in the recrawl queue indefinitely.
- Infinite calendars and paginated archives that generate URLs forever.
- Duplicate content clusters where twenty URLs serve one page.
None of these are exotic. They are the default behavior of most ecommerce platforms and CMSs unless someone deliberately reins them in.
A quick way to size the problem: compare the number of URLs your CMS reports as published against the number of distinct URLs a full crawl discovers. On a healthy site those numbers are close. When a 4,000-product catalog crawls out to 190,000 URLs, the gap is your leak, and it is almost always faceted navigation plus sorting parameters multiplying against each other.
Crawl budget optimization: the fixes that work
Ordered roughly by return on effort, here is a working crawl budget optimization guide:
- Make the server faster. Cut time-to-first-byte, cache aggressively, and eliminate 5xx spikes. Capacity is the ceiling on everything else.
- Block genuinely worthless URL patterns in robots.txt. Internal search results, cart and checkout paths, filter combinations you never want crawled.
- Return correct status codes. Gone means 410. Missing means 404. Do not serve 200 with a “no results” message.
- Flatten redirect chains. Point every old URL directly at the final destination, not at another redirect.
- Fix internal linking. Pages you care about should be reachable in three clicks from the homepage, and linked from more than one place.
- Keep XML sitemaps clean. Only canonical, indexable, 200-status URLs, with accurate
lastmoddates. A sitemap full of redirects and 404s teaches Google to trust it less.
Two things that do not work. Crawl-delay in robots.txt is ignored by Google entirely — it is a directive other crawlers honor, not Googlebot. And you cannot increase your allocation by pinging, resubmitting sitemaps repeatedly, or adding priority values. There is no lever labeled “how to increase crawl budget” beyond making your site faster and more worth crawling.
robots.txt versus noindex for crawl control
This distinction trips up experienced people, so be precise about it.
Robots.txt controls crawling. A disallowed URL is never fetched, which saves the request. But because it is never fetched, Google cannot see any tag on the page — including a noindex. That is why Noindex: in robots.txt does not work; Google confirmed it was never officially supported and stopped honoring it in 2019.
Meta robots and the X-Robots-Tag header control indexing. They require the page to be crawled to take effect.
<meta name="robots" content="noindex, follow">So the sequencing matters: to remove a page already in the index, let it be crawled with a noindex until it drops, then consider blocking it. Block first and you strand it — indexed, uncrawlable, unfixable. That specific mistake is what produces the “Indexed, though blocked by robots.txt” warning, and it is the most common self-inflicted wound in this whole area.
For pages you never want crawled in the first place — checkout flows, account pages, faceted permutations that were never going to rank — robots.txt is the right tool and saves the request entirely. For pages that must stay accessible to users but out of the index, use noindex and accept the crawl cost. Picking the wrong one of those two is a much bigger deal than any tuning you can do afterwards.
Measuring it: what to actually look at
The Crawl Stats report in Search Console (under Settings) is the primary source. It shows total requests per day, average response time, and a breakdown by response code, file type, and purpose — discovery versus refresh. Three readings matter: a rising average response time means you are throttling yourself; a large share of 3xx or 4xx responses means waste; and a discovery-heavy split on a mature site means the crawler is finding new URLs faster than you are creating real pages.
Server log files are the ground truth. They tell you exactly which URLs Googlebot fetched, how often, and what you returned. If the top twenty most-crawled paths on your site are filter permutations, you have your answer without further analysis.
Alongside logs, run a full crawl of your own so you can compare what exists against what gets fetched. SEO Rocket’s technical audit crawls entire sites — verified past 900 pages — and reports evidence per issue rather than a total: the actual redirect chains, the actual duplicate title clusters, the actual URLs returning the wrong status. Its PageSpeed and Core Web Vitals view pairs lab metrics with real-user field data, which is the honest way to judge whether your server is fast enough to sustain a higher crawl rate. It will not rewrite your robots.txt or deploy redirects for you — that stays with your team.
Crawl budget best practices for large sites
A workable standing policy, once the cleanup is done. Keep one canonical URL per piece of content and enforce it with both tags and redirects. Decide deliberately which faceted combinations deserve indexing — usually the ones with real search demand — and cut off the rest. Retire dead sections properly with 410s instead of leaving them to rot. And watch response time as a first-class SEO metric, not just a UX one.
Expect this to be slow. After a large cleanup, Googlebot reallocates over weeks, not days, and the effect shows up as more of your important URLs being refetched sooner rather than as an immediate ranking jump. Judge it on crawl stats and indexation coverage first; traffic follows later, if the pages you freed up were worth indexing in the first place.
