Crawl Budget: When It Matters and When It Does Not

01

Identified claim

A common explanation says:

“Google is not indexing this page because the site ran out of crawl budget.”

For many small sites, that explanation is disproportionate.

Google’s crawl-budget guide is advanced guidance primarily for very large sites, rapidly changing sites, and sites with a substantial portion of URLs classified as “Discovered – currently not indexed.”^[5] Google gives rough examples such as sites with about one million frequently changing pages or sites with about 10,000 pages changing daily. Those are classification examples, not exact gates. Optimize your crawl budget^[1]

02

Sources and evidence

What crawl budget means

Google defines crawl budget through two main elements:

• Crawl capacity limit: How much crawling a hostname can handle without being overloaded.
• Crawl demand: How much Google wants to crawl the site and its URLs.

The practical budget is the set of URLs Google can and wants to crawl. A site may have unused server capacity while demand remains low. It may have high demand while server instability forces Google to slow down.

Google treats hostnames separately for this purpose. A main domain and a subdomain can have different crawl conditions. Optimize your crawl budget^[1]

Who should care

Crawl-budget work is more likely to be justified when:

• the site contains hundreds of thousands or millions of URLs;
• inventory changes quickly;
• a large share of URLs remains discovered but not crawled;
• Googlebot regularly encounters 429 or 5xx responses;
• faceted navigation creates enormous duplicate spaces;
• calendars or parameter systems generate near-infinite URLs;
• a migration requires large-scale reprocessing;
• logs show that important new URLs wait while low-value variants are repeatedly fetched.

A 70-page publication with healthy same-day crawling does not need a crawl-budget task force. It needs accurate sitemaps, crawlable links, stable hosting, and useful pages.

A practical decision table

Crawl capacity can shrink

Google reduces crawl rate when a host returns significant numbers of server-overload responses such as 500, 503, or 429^[4]. The slowdown affects the hostname, not only the failing URLs. Once errors fall, crawl rate can recover. Reduce Google crawl rate^[3]

Repeated failures can therefore create a real capacity problem:

New URLs published
        ↓
Googlebot increases requests
        ↓
Origin returns 429 and 503
        ↓
Crawler slows across hostname
        ↓
Fresh inventory waits longer

The fix is stable serving capacity, not an indexing request for each page.

Crawl demand can remain low

Even when a server can handle more traffic, Google may not want to crawl every URL frequently.

Demand is influenced by factors such as:

• URL popularity;
• freshness and change rate;
• overall value and uniqueness;
• stale or duplicate inventory;
• site moves and reprocessing events;
• Google product needs.

Crawling more often is not itself a ranking signal. Google’s crawling myths documentation explicitly rejects the claim that higher crawl rate improves search positions. Myths and facts about crawling^[2]

Wasteful URL inventory

The most controllable problem is often not a fixed budget number but the number of crawlable URLs the site creates.

Common waste includes:

• filter combinations;
• alternate sort orders;
• session and tracking parameters;
• internal links to redirects;
• calendar pages for unlimited dates;
• duplicate print and preview routes;
• empty search results;
• protocol and hostname variants;
• unstable pagination;
• API URLs linked as pages.

Use the correct control for each family:

Google’s current crawl-budget guidance advises using robots.txt for URLs that should not be crawled at all and warns that noindex still requires a crawl before the page is dropped. It also notes that freeing crawl capacity does not guarantee that the saved requests will automatically be spent elsewhere unless the host was already at its capacity limit. Optimize your crawl budget^[1]

Caching and unchanged pages

Support conditional requests and 304 Not Modified where appropriate. When a resource has not changed, a 304 can reduce transfer and processing costs while allowing Google to reuse the cached version. Optimize your crawl budget^[1]

Use meaningful lastmod values in sitemaps. Do not update every timestamp on every build if the page did not materially change.

What to inspect

For a site large enough to justify the work, combine:

• Crawl Stats;
• server logs;
• Page Indexing reasons;
• sitemap inventories;
• response-time and error metrics;
• URL-pattern counts;
• internal-link generation;
• change frequency;
• Googlebot request distribution.

Do not diagnose crawl budget from one screenshot or one excluded URL.

03

Conclusion

The claim that every indexing delay is a crawl-budget problem is false.

Use this decision sequence:

Is the page already crawled? If yes, crawl budget is not the direct reason it was not indexed.

Is the site small and normally crawled promptly? If yes, investigate the exact URL rather than budget.

Is a large share of important inventory persistently discovered but not crawled? Continue.

Do logs show low-value URL families consuming requests? Reduce that inventory.

Do 429 or 5xx responses cause hostwide slowdown? Restore capacity.

Do important pages lack links or accurate sitemap entries? Repair discovery.

Does the site change rapidly enough to require frequent recrawling? Prioritize stable canonical inventories and accurate modification signals.

A concise operating rule is:

Treat crawl budget as a large-site capacity-and-inventory problem, not a universal explanation for one missing page.

04

Limitations

Google does not provide a fixed daily crawl allowance or a button that reallocates crawling to preferred URLs.

The published site-size examples are rough. A smaller site with severe URL explosion or hosting instability can have meaningful crawl problems, while a large stable site can be crawled efficiently.

Improving crawl efficiency can help Google reach important URLs. It does not guarantee indexing, rankings, or that every saved request is immediately reassigned.

The review is complete when the site can explain:

• whether capacity or demand is the limiting side;
• which URL families receive crawler attention;
• whether serving errors cause slowdown;
• whether important inventory is linked and submitted;
• which duplicate spaces can be consolidated or blocked;
• whether the site is actually large or volatile enough for crawl-budget work.

References

Sources behind this record

1. Optimize your crawl budget — Google (accessed July 26, 2026)
2. Myths and facts about crawling — Google (accessed July 26, 2026)
3. Reduce Google crawl rate — Google (accessed July 26, 2026)
4. How HTTP status codes affect Google's crawlers — Google (accessed July 26, 2026)
5. Page indexing report — Google (accessed July 26, 2026)

Corrections

Correction history

No corrections recorded.

To report an error, use the public corrections path.