What Is Crawl Budget and When Should You Optimize It?

What Is Crawl Budget, Exactly?

Crawl budget combines two things Google tracks separately. Crawl capacity limit is how many requests per second Googlebot can make to your server without causing problems, based on how your server responds under load. Crawl demand is how much Google actually wants to crawl your site, based on the site's overall quality, freshness of content, and popularity.

Together, these determine roughly how many URLs Googlebot will request from your site in a given day. If your site has fewer URLs than that number, crawl budget is not a limiting factor. If your site has far more URLs than Googlebot is willing to crawl, some pages, often your newest or least linked ones, may go uncrawled for extended periods.

Does Crawl Budget Matter for Small Websites?

Rarely. Google's own documentation states plainly that crawl budget is generally not something site owners need to actively manage unless their site has many thousands of URLs, or unless the server itself struggles under Googlebot's crawl requests. A typical business site with a few dozen to a few hundred pages will usually be crawled completely and efficiently without any intervention.

The exception is a slow, overloaded server. If your hosting cannot handle Googlebot's requests without slowing down or timing out, Google will reduce its crawl rate to avoid causing problems, and that reduction can affect even a small site.

What Wastes Crawl Budget?

Common causes of wasted crawl budget on larger sites

Cause	Why it wastes budget
Faceted navigation / filter URLs	Can generate thousands of near-duplicate URL combinations from a handful of real pages
Soft 404s	Pages that return a 200 status but show no real content still get crawled repeatedly
Redirect chains	Each hop in a chain consumes a separate crawl request before reaching the final URL
Duplicate content from cannibalization	Google crawls both competing pages instead of consolidating attention on one
Orphaned low-value pages still in the sitemap	Old campaign pages or test pages that were never removed keep getting crawled

How Do You Check Crawl Stats in Search Console?

Search Console's Crawl Stats report, found under Settings, shows total crawl requests over the last 90 days, broken down by response code, file type and crawl purpose. A rising trend of 404s or 5xx responses in that report is usually the clearest early sign of a crawl budget problem developing.

The Pages report under Indexing shows a separate but related signal, the count of pages Google has "Discovered, currently not indexed," which often points to pages Google found but chose not to prioritize crawling or indexing yet.

What Actually Fixes Crawl Budget Problems?

Reducing the number of low-value URLs Google has to choose between is the real fix, not trying to increase the budget itself, since Google does not offer a direct way to raise it. That means canonicalizing or parameter-handling faceted navigation URLs, fixing redirect chains down to a single hop, removing or noindexing genuinely low-value pages, and resolving the kind of duplicate content covered on our keyword cannibalization guide.

A clean, accurate XML sitemap that only includes canonical, indexable URLs also helps Google prioritize correctly, since a sitemap full of redirected or noindexed URLs sends a confusing signal about what actually matters on the site.

Does Blocking Pages in Robots.txt Save Crawl Budget?

Partially, and it is often misunderstood. Robots.txt prevents Googlebot from requesting a URL at all, which does redirect crawl attention elsewhere. What it does not do is guarantee the URL stays out of the index, since Google can still index a URL it has never crawled if other pages link to it, showing it in results with no description.

For pages you want both uncrawled and unindexed, a noindex meta tag on a crawlable page, or removing the URL and its internal links entirely, is more reliable than robots.txt alone.

Frequently Asked Questions