The robots txt crawl delay directive asks a crawler to wait a set number of seconds between requests. It is one of the oldest extensions to the Robots Exclusion Protocol, it appears in nearly every generated robots.txt file, and Google has never supported it. That last part surprises people who have been shipping crawl-delay lines for a decade.
The directive is not useless — Bing genuinely honors it — but the situations where it is the right answer are narrower than the advice suggests, and the cost of using it is higher than most people realize.
The Syntax
A crawl delay robots txt rule sits inside a user-agent group and takes a number of seconds:
User-agent: bingbot
Crawl-delay: 5
That asks Bingbot to leave five seconds between requests to your host. Applied to a wildcard group, it targets any crawler that supports it and does not have a more specific group of its own.
Do the arithmetic before you ship it. A five-second delay caps crawling at roughly 17,000 URLs per day in the best case. On a 200,000-URL ecommerce catalog, that is nearly twelve days for a single full pass, assuming the crawler never pauses — which it will. Recrawl frequency for your most important pages drops accordingly.
Google Ignores It. Completely.
Googlebot does not read Crawl-delay. This was informal for years and became official on 2 July 2019, when Google announced it was dropping support for unsupported robots.txt rules — crawl-delay, noindex, and nofollow among them — as part of standardizing the protocol. Enforcement began 1 September 2019.
Nothing warns you. The line parses as an unknown directive and is discarded silently. If Googlebot is hammering your server and you added a crawl-delay to stop it, you changed nothing at all.
Google’s stated reasoning is that its crawl rate is already adaptive. Googlebot monitors server response times and error rates, and backs off automatically when a host starts responding slowly or returning 5xx errors. In practice that feedback loop works — but it works on a delay, which is cold comfort during an active incident.
Who Does Honor It
Bingbot supports crawl-delay and documents it. Yandex historically supported it, though it now prefers its own controls. Many smaller crawlers, SEO tools, and archive bots respect it, and those are frequently the ones causing real load — a third-party crawler with no rate limiting can generate more requests per second than every search engine combined.
Bad actors, of course, honor nothing. Scrapers and content thieves ignore robots.txt entirely, and if they are the source of your load the only real answers are rate limiting at the edge, firewall rules, or a bot-management service. No directive in a text file will slow down a crawler that never reads it.
So a crawl-delay line is not wasted. Just be precise about who you are targeting, and do not let it live in a wildcard group where it silently throttles crawlers you actually want moving quickly.
What to Do About Googlebot Instead
If Googlebot’s crawl rate is genuinely a problem, you have three real options, in ascending order of preference.
- Search Console crawl controls. Google retired the old manual crawl rate limiter tool in January 2024 and now leans on automatic rate adjustment, so the practical path is the Crawl Stats report plus a support request for genuinely severe cases.
- Return 503 or 429 responses. Serving
503 Service Unavailablewith aRetry-Afterheader tells Googlebot to back off fast — usually within minutes. This is the correct emergency brake. Keep it brief; sustained 5xx responses over days will cause URLs to drop out of the index. - Fix the capacity problem. Unglamorous and correct. A server that a search crawler can overwhelm will also fold under a product launch, a newsletter send, or a modest traffic spike.
Never block Googlebot with robots.txt to reduce load. It is a permanent, indexing-level change to solve a temporary performance problem, and recovering from it takes far longer than the incident would have.
What Crawl-Delay Costs You
Throttling is not free, and the bill arrives on a lag. Slower crawling means slower discovery of new pages, slower reprocessing of updated ones, and slower recognition of redirects and removals. On a news or ecommerce site where freshness matters, that is a direct revenue cost.
It also interacts badly with large sites. Google allocates crawl capacity partly on how fast and reliably your server responds; a host that answers quickly earns more crawling. Artificially spacing requests does not improve that signal — it just reduces throughput while your server sits idle between fetches.
Where a delay is genuinely justified is narrow: a small shared-hosting site being crawled aggressively by a dozen third-party bots at once, where the cost of slow discovery is trivial and the cost of downtime is not. Outside that case, a targeted Disallow on the expensive path beats a blanket delay every time, because it removes the requests instead of merely spreading them out.
The Usual Root Cause Is Not the Crawler
When crawl volume overwhelms a site, the crawler is rarely the real issue. The site is generating far more URLs than it has content. Faceted navigation is the classic offender: five filters with six values each produce tens of thousands of unique URLs from one category page, and a crawler will dutifully try to fetch all of them.
Other reliable sources of URL explosion:
- Session IDs or tracking parameters appended to internal links
- Calendar interfaces that generate a URL for every date, forever
- Internal search results linked from anywhere on the site
- Sort and pagination parameters combined with filters
The pattern to look for in logs is a crawler spending most of its requests on URLs that carry parameters. If 70 percent of Googlebot’s fetches are hitting filter combinations, you do not have a crawl-rate problem — you have a URL generation problem wearing a crawl-rate costume, and throttling it just makes the same waste take longer.
Cut the URL space and crawl pressure falls without throttling anything. Canonical tags, parameter-level disallows targeted at genuinely infinite spaces, and removing internal links to filtered variants do more in a week than a crawl-delay does in a year.
Diagnose Before You Throttle
Server logs settle the argument in an afternoon. Filter by user agent and count requests per hour, then check which paths absorbed them. Two patterns show up repeatedly: the “Googlebot” traffic is largely spoofed bots faking the user agent — verify with a reverse DNS lookup — or a single low-value directory is consuming the majority of requests.
Search Console’s Crawl Stats report gives the same picture from Google’s side, broken down by response code, file type, and purpose. If average response time is climbing while crawl requests stay flat, your server is the constraint, not the crawler.
From the site side, a full crawl of your own tells you how many URLs actually exist versus how many you meant to publish. SEO Rocket’s deep crawls, verified past 900 pages, report per-issue evidence with the real URLs, titles, and H1 text involved — which is usually where a parameter explosion becomes obvious. It does not edit your robots.txt or server config; those remain deliberate human changes. But it will show you, concretely, that the problem was 40,000 filter combinations rather than an impatient crawler.
