Most SEO advice about crawl budget is guesswork dressed up as strategy — someone reads a crawl-stats chart in Search Console, sees a number dip, and invents a story. Log file analysis is the one technique that replaces the story with a fact. Your server logs record every request Googlebot ever made to your site: which URL, at what second, with what status code, how many bytes. Nothing is sampled, nothing is estimated, nothing is filtered by a JavaScript layer. If you want to know what a crawler is doing instead of what a tool guesses it’s doing, this is the only source of ground truth you own outright.
Why the log is the only unfiltered source
Every other window into crawling is second-hand. Search Console’s Crawl Stats report is aggregated, rounded, and delayed by a day or two. Third-party crawlers simulate Googlebot; they don’t observe it. Analytics never sees a bot at all because bots don’t fire the JavaScript that analytics depends on. The server log is different — it’s written by your own web server the instant a request lands, before any caching, rendering, or tag manager gets involved. When Googlebot hits a URL, your server writes a line. That line is evidence.
The practical payoff: log file analysis is how you catch problems that no crawl audit can surface. A site crawler can tell you a page returns a 200 and has a good title. Only the log can tell you Googlebot has requested that page twice in ninety days while hammering a faceted-navigation URL six hundred times. The first is a page-quality question. The second is a crawl-economics question, and it’s usually the one killing your indexing.
What a single log line actually contains
Most servers write the Combined Log Format. A single line looks dense but decodes cleanly into the fields that matter for SEO:
- IP address — the origin of the request, and the thing you must verify before trusting the user-agent.
- Timestamp — down to the second, which lets you reconstruct crawl sessions and spot bursts.
- Request path and method — the exact URL fetched, including query strings, which is where crawl waste hides.
- HTTP status code — 200, 301, 404, 503; the single most diagnostic field in the line.
- Bytes transferred — a proxy for page weight and for how much of your crawl budget each fetch consumes.
- User-agent string — which claims to be Googlebot, and frequently lies.
That last point is not optional paranoia. Scrapers, competitors, and malware routinely forge the Googlebot user-agent to bypass rate limits or dodge blocks. Before you draw a single conclusion, verify authenticity: run a reverse DNS lookup on the IP, confirm it resolves to a googlebot.com or google.com host, then do a forward lookup back to the same IP. Alternatively, match against Google’s published crawler IP ranges. Skip this step and you’ll optimize your crawl budget around traffic from a Ukrainian scraper. I’ve watched teams do exactly that.
The crawl-budget mechanism you’re actually diagnosing
Log file analysis only makes sense once you understand what you’re looking for. Googlebot’s crawling is governed by two forces. The first is crawl capacity — how many simultaneous requests your server can handle before response times degrade. If your pages start answering slowly or returning 5xx errors, Google backs off to avoid taking your site down. The second is crawl demand — how much Google wants your pages, driven by popularity, freshness, and perceived quality. A page nobody links to and Google rates as thin generates almost no demand.
Your effective crawl budget is the smaller of the two. The whole game is making sure the budget you have lands on the URLs that earn revenue, not on parameter permutations, session IDs, infinite calendars, and dead redirect chains. Logs are where you see the misallocation directly: the ratio of crawler hits on your money pages versus the junk. Fix that ratio and pages that were being ignored start getting indexed within a crawl cycle or two.
Getting hold of the logs
Where your logs live depends on your stack. On a traditional Apache or nginx box they’re in /var/log/. Behind Cloudflare or Fastly you’ll pull them from the CDN’s log-export feature, not the origin, because the CDN absorbs a huge share of bot traffic the origin never sees. On managed WordPress hosts you request them from support or a dashboard. On AWS or GCP they’re in load-balancer or CloudFront/Cloud CDN logs sitting in object storage. Wherever they are, insist on at least thirty days of retention — a seven-day window can’t reveal a monthly crawl rhythm, and migrations need the before-and-after both captured. Watch two things: the files get large fast (compress and sample if needed), and raw logs contain user IP addresses, which is personal data you must handle under your privacy obligations.
The five questions to ask first
Don’t open a log file and wander. Interrogate it with a fixed set of questions, in order:
- Which URLs get crawled most? Sort requests by path descending. If your top ten are parameterized junk or paginated tag archives, you’ve found your leak.
- Which important pages get crawled rarely or never? Cross-reference your key URLs against the log. A page Googlebot hasn’t touched in sixty days is effectively invisible.
- What’s the status-code distribution? A healthy site is overwhelmingly 200s and 304s. A rising share of 404s, 301 chains, or 5xxs is budget set on fire.
- How is crawl split by site section? Group hits by directory. If your blog gets 70% of crawl attention while your product pages starve, your internal linking is misdirecting the bot.
- Mobile vs desktop Googlebot? Since mobile-first indexing, the smartphone crawler should dominate. If desktop still leads, that’s a signal worth chasing.
A worked micro-example
Here’s the shape of a real diagnosis, with illustrative numbers. Say a 4,000-page ecommerce site pulls a 30-day log and finds Googlebot made roughly 120,000 requests. Break it down: 46% went to URLs containing ?color= and ?sort= parameters — faceted navigation the site never wanted indexed. Another 11% hit a 301 redirect that itself pointed to a second 301 before resolving. Meanwhile, 300 core product pages received fewer than three crawler visits each in the entire month.
The read is unambiguous. Over half the crawl budget is being spent on filter combinations and a lazy redirect chain, and the pages that make money are starving as a result. The fixes follow directly: disallow the parameter patterns in robots.txt, collapse the redirect chains to a single hop, and strengthen internal links to the neglected products. On sites like this the neglected pages typically start getting crawled within one to two cycles, and indexing follows. You could never have found this from a crawl simulator — the simulator doesn’t know Google was wasting 57% of its attention.
The patterns that signal real problems
Once you’ve run the questions a few times, certain shapes jump out. A sudden crawl collapse — hits dropping to near zero overnight — almost always means a robots.txt mistake or a server outage during Google’s visit; check the timestamp against your deploy log. A 404 spike concentrated on one path pattern usually traces to broken relative URLs shipped in a template. A slow rise in response time paired with a fall in crawl volume is Google throttling itself because your server got slower — a capacity problem, not a demand one. Orphan URLs that still draw crawler hits are old pages nothing links to internally but Google remembers; they quietly drain budget. And a low ratio of 304 Not Modified responses on stable pages means you’re not sending proper caching headers, forcing full re-fetches of content that never changed.
Tools, from a grep pipe to a data warehouse
You don’t need enterprise software to start. For a single site and a manageable log, a few command-line pipes — grep to isolate Googlebot lines, awk to extract fields, sort and uniq -c to rank them — answer most of the five questions in minutes, and GoAccess gives you a free visual dashboard on top. Step up and Screaming Frog‘s Log File Analyser or JetOctopus join log data to crawl data so you can see, in one view, which crawled URLs are also orphaned or non-indexable. At true scale — millions of lines a day, multiple properties — you’re loading logs into BigQuery, Elasticsearch, or Splunk and querying them like the datasets they are.
The highest-leverage move is joining the log to a real crawl of your site, because that’s where crawl behaviour meets page reality. SEO Rocket’s site audit runs a genuine crawler over your pages and surfaces the technical issues — redirect chains, orphan pages, broken internal links, indexability blocks — that the log then confirms Google is actually spending budget on. Reading the two side by side turns “Googlebot ignores this section” from a mystery into a fix list.
Turning findings into changes
Analysis with no action is a hobby. Prioritize what the log surfaces by damage: kill crawl waste first (block parameters, fix redirect chains, remove infinite spaces), because reclaimed budget flows straight to real pages; next, get genuinely important but under-crawled URLs linked and submitted; then chase 5xx errors, which signal capacity limits Google is actively backing away from. Run the pass monthly for a stable site and weekly through any migration or major release, when crawl patterns shift fastest. Keep each month’s baseline so you’re reading a trend line, not a single snapshot — one anomalous day means nothing, a three-month drift means everything. Feed the findings into rank tracking too: when a starved section starts getting crawled, watch whether its rankings and impressions follow, because that’s the proof your log file analysis translated into results.
The honest limits of log data
Logs are ground truth about crawling, and nothing else. They tell you what Googlebot fetched — they cannot tell you whether Google indexed it, ranked it, or judged the content any good. A page can be crawled daily and still sit on page five because it’s thin or duplicative; the log will never explain why. Logs also miss rendered JavaScript behaviour, off-server CDN hits you didn’t export, and any signal about link value or user satisfaction. Treat log file analysis as the crawl-and-technical layer of a fuller diagnosis, cross-checked against Search Console’s index coverage and GA4 as ground truth for what actually earns traffic. Anyone selling logs as a complete SEO answer is overselling. This methodology is part of the same playbook proven across 1,000,000+ ranking pages: fix the technical foundation so quality content gets the crawl attention it deserves — logs just tell you exactly where the foundation is cracked.
Frequently asked questions
How often should I do log file analysis?
Monthly is the right cadence for a stable site — that captures a full crawl rhythm without drowning you in data. Move to weekly during a migration, a redesign, or after any change to robots.txt or site architecture, because those are exactly the moments crawl behaviour shifts and a mistake costs the most. Keep every month’s numbers so you can compare against a baseline rather than reacting to a single day.
Can I do log file analysis without technical skills?
Partly. Tools like GoAccess or Screaming Frog’s Log File Analyser give you a visual dashboard without any command line, so the five diagnostic questions are answerable by anyone who can filter a spreadsheet. Getting the logs in the first place, verifying Googlebot authenticity, and handling large files at scale do lean technical — that’s usually where a developer or a consultant earns their keep.
How much log history do I need?
Thirty days is the practical minimum, because Google’s crawl of a mid-size site follows a monthly rhythm and a seven-day window can’t show it. For migrations, keep the full before-and-after period so you can prove the change helped or catch it if it hurt. Ninety days is better still for spotting slow drifts in crawl volume that a single month would hide.
Is it worth doing for a small site?
For a 50-page brochure site, rarely — crawl budget isn’t a constraint when Google can fetch your whole site in a minute. It becomes essential once you cross a few thousand URLs, run faceted navigation, or generate pages programmatically, because that’s when crawl budget turns scarce and misallocation starts costing you indexed pages.
