Almost everyone who reaches for the robots.txt crawl-delay directive is trying to solve the wrong problem. They see Googlebot hammering their server in the logs, they panic, they add Crawl-delay: 10, and they wait for relief that never comes — because the single most important crawler on the web has never read that line and never will. Crawl-delay is a real directive with a real effect, but it aims at a target most sites aren’t actually facing. Before you throttle anything, you need to know exactly what this line does, who obeys it, and whether the crawler is even your bottleneck.
What robots.txt crawl-delay actually tells a crawler
The crawl-delay directive is a non-standard line in your robots.txt file that asks a crawler to wait a set number of seconds between requests to your site. The syntax is trivial:
User-agent: *
Crawl-delay: 10
Read literally, that says “wait ten seconds between fetches.” It was never part of the original robots exclusion protocol — it was a convention some search engines adopted independently, which is exactly why support for it is so inconsistent. There is no governing spec that defines whether the number means seconds, a rate, or a priority bucket, so different crawlers have historically interpreted it differently. Bing treats the value as literal seconds. Yandex historically read it the same way. That ambiguity alone is a warning sign: a directive nobody standardized is a directive you can’t rely on.
The throughput math nobody runs before adding it
Here is the calculation that should stop most people from ever using crawl-delay. A day has 86,400 seconds. If you set Crawl-delay: 5, you are telling the crawler it may fetch at most one URL every five seconds — roughly 17,000 URLs per day, and that’s a ceiling, not a target. Set it to 30 and you’ve capped a compliant bot at under 3,000 fetches a day.
For a 40-page brochure site, that’s irrelevant. For a 200,000-URL ecommerce catalog, a five-second delay means a full recrawl would take almost two weeks even in a perfect world — and crawlers don’t work in a perfect world, so in practice it’s slower. You’ve just told search engines to discover your new products and price changes at a crawl. The directive that was supposed to protect your server has quietly throttled your own indexing. Run this number for your URL count before you ever type the line.
Why Google ignores robots.txt crawl-delay completely
This is the part that catches people out: Googlebot does not support crawl-delay and never has. Google formalized this in 2019 when it announced it would stop honoring unsupported robots.txt directives (crawl-delay, noindex, and nofollow in robots.txt) as of September 1 of that year. Nothing changed for crawl-delay in practice — Googlebot was already ignoring it — but the announcement made the position official and unambiguous.
Google’s reasoning is mechanical, not stubborn. Instead of reading a static number you picked, Googlebot sets its own crawl rate dynamically based on your server’s observed health — response times, error rates, and connection behavior. If your server starts slowing down or returning server errors, Googlebot backs off on its own. If your server is fast and healthy, it crawls faster. Google calls the ceiling side of this your “crawl capacity limit,” and it’s adaptive per host. A fixed ten-second delay would be worse at protecting your server than Googlebot’s own real-time signal — which is precisely why it doesn’t offer you the lever.
Who still honors crawl-delay
The directive isn’t dead — it’s just not for Google. The crawlers that respect a robots.txt crawl-delay include:
- Bing — Bingbot honors crawl-delay and reads the value as seconds between requests. Bing Webmaster Tools also exposes a “Crawl Control” schedule for finer control by time of day.
- Yandex — historically honored it, though Yandex has since steered webmasters toward a crawl-rate setting in its own Webmaster console rather than the robots.txt line.
- Many third-party and SEO crawlers — tools like Ahrefs’ and Semrush‘s bots, and various scrapers, will generally respect it. If an aggressive third-party bot is your actual problem, crawl-delay can genuinely help.
So the honest use case is narrow: crawl-delay is a tool for managing Bing and well-behaved non-Google bots, not for reining in Googlebot. If your log files show Google as the heavy hitter, this directive is not your answer.
The real question: crawl demand vs. crawl capacity
Here’s the framework that replaces the panic reaction. A crawler’s behavior on your site is the product of two things: crawl demand (how much it wants to crawl, driven by your site’s size, freshness, and popularity) and crawl capacity (how much your server can handle before it degrades). Excessive crawling is almost always a demand problem wearing a capacity costume.
When people see a crawler making tens of thousands of requests a day, they assume the bot is misbehaving and try to cap capacity with crawl-delay. But a well-designed crawler only crawls that hard because your site is offering that many URLs to crawl. Throttle the bot and the underlying URL sprawl is still there — you’ve just slowed the symptom. Fix the demand side and the aggressive crawling disappears on its own, with no directive required.
The usual root cause is URL explosion, not an aggressive bot
In practice, a crawler chewing through your budget is nearly always a sign your site is generating far more URLs than it has real pages. The classic sources of this sprawl:
- Faceted navigation — filter combinations (color + size + brand + price) that multiply into thousands or millions of crawlable parameter URLs.
- Session IDs and tracking parameters appended to URLs, so every “page” exists in dozens of near-identical variants.
- Calendar and pagination traps — infinite “next month” links or deep paginated archives that never bottom out.
- Sort orders and internal search results that create a fresh URL for every permutation.
A worked example: say a mid-size store has 8,000 real products. Add a faceted filter with five filter types averaging four options each, all crawlable, and you can generate hundreds of thousands of URL combinations from those same 8,000 pages. Googlebot dutifully tries to crawl them, your logs light up, and it looks like an aggressive-bot problem. It isn’t. Adding crawl-delay would slow discovery of your actual products while doing nothing about the combinatorial explosion. The fix is to stop generating the junk URLs — block parameter paths in robots.txt, add rel="canonical" to the clean version, use noindex where appropriate, and tighten internal linking so you’re not feeding crawlers links to the sprawl in the first place.
How to actually control Googlebot
If Googlebot genuinely is overloading a healthy server — rare, but it happens on very large or newly expanded sites — you have real options, none of which is crawl-delay:
- Return 503 or 429 with a Retry-After header. This is Google’s own recommended lever. A short burst of
503 Service Unavailableor429 Too Many Requestsresponses tells Googlebot to slow down fast, and it will. Keep it to hours or a day or two — sustained 5xx errors for weeks can get URLs dropped from the index. - Fix the underlying capacity. Caching, a CDN, and faster response times raise your crawl capacity limit, which lets Googlebot crawl efficiently without straining anything.
- Report it to Google as a last resort. Google retired the manual crawl-rate limiter tool in Search Console in early 2024 precisely because the adaptive system handles this better; if it truly can’t, Google provides a report form for excessive-crawling emergencies.
A decision rule for when crawl-delay is the right tool
Strip away the noise and the choice is simple. Use crawl-delay only when all of these are true: the offending bot is Bing or a third-party crawler (not Google), your server is genuinely straining, and you cannot fix the capacity or the URL sprawl quickly. In that narrow window, a modest value like Crawl-delay: 2 or 5 for the specific user-agent buys you breathing room. In every other case — and that’s most cases — the honest move is to fix crawl demand or server capacity, and leave the directive out entirely. A robots.txt crawl-delay aimed at Google is worse than useless: it does nothing to Google and throttles the honest bots you actually wanted crawling you.
Diagnose before you throttle
The one step that separates people who fix this from people who guess at it is log analysis. Your server access logs tell you exactly which user-agents are requesting what, how often, and — critically — which URL patterns they’re burning requests on. Nine times out of ten the answer jumps out: a single parameter combination or a faceted path eating 70% of the crawl budget. That’s a URL-sprawl fix, not a crawl-delay fix.
This is where a real-crawler site audit earns its keep. SEO Rocket’s site audit crawls your site the way Googlebot does and surfaces the structural problems that drive runaway crawling — parameter bloat, thin duplicate URLs, orphaned pages, and broken canonical signals — so you’re treating the disease instead of the symptom. Pair that with rank tracking and AI-visibility monitoring and you can confirm that cleaning up crawl demand improved indexing and rankings, rather than hoping a directive Google ignores did something. That diagnose-first discipline is drawn straight from a playbook proven across 1,000,000+ ranking pages: never throttle a crawler until you’ve proven the crawler is the problem.
Frequently asked questions
Does Google respect robots.txt crawl-delay?
No. Googlebot has never supported the crawl-delay directive and Google made this official in 2019. Googlebot sets its own crawl rate based on your server’s health and your site’s crawl demand, so adding crawl-delay has zero effect on Google. Use 503/429 responses with Retry-After if you genuinely need to slow Googlebot.
What crawl-delay value should I use for Bing?
There’s no universal number — it depends entirely on your server capacity and URL count. A value of 1 to 5 seconds is a reasonable starting point for a healthy site. Remember the math: Crawl-delay: 5 caps Bingbot at roughly 17,000 URLs a day, so on a large site an aggressive value can starve your own indexing. Start low and only raise it if your logs show real strain.
Will crawl-delay hurt my SEO?
It can. Because the directive slows how fast compliant crawlers discover and refresh your content, a high crawl-delay on a large or frequently updated site delays indexing of new and changed pages. Since Google ignores it anyway, you get all of that downside on Bing and other honest bots with none of the intended benefit against Googlebot.
What should I use instead of crawl-delay for Google?
Fix the root cause first: block junk parameter URLs, canonicalize duplicates, and improve server response times to raise your crawl capacity limit. If Googlebot is still overwhelming a healthy server, return 503 or 429 responses with a Retry-After header for a short period — that’s Google’s own recommended way to ask Googlebot to back off.
