Crawl budget only becomes a problem when it is already costing you traffic. Here is how to spot it before it does.
What crawl budget actually is
Crawl budget is the number of URLs Googlebot is willing to request from your site in a given period. For a small brochure site it is effectively unlimited and not worth thinking about. For an ecommerce catalogue with faceted navigation, or a publisher with fifteen years of archives, it is one of the first things we check — because when it runs out, your newest and most valuable pages are the ones that go unseen.
1. Discovered — currently not indexed is growing
Open the Page Indexing report in Search Console. If the “Discovered — currently not indexed” bucket is large and trending upward, Google knows those URLs exist and has chosen not to spend the budget fetching them. That is a direct statement about how it values your site relative to the cost of crawling it.
2. New pages take weeks to appear
Publish a page and watch how long it takes to get indexed without manual submission. On a healthy site of moderate authority, that is hours to a couple of days. If it is consistently two weeks or more, Googlebot is not returning to your site often enough, or it is spending its visits somewhere unproductive.
3. Your URL count is far larger than your page count
Run a crawl and compare the number of URLs found against the number of genuinely distinct pages you believe you have. A tenfold difference almost always means faceted navigation, session parameters, sort orders or calendar pagination are generating near-infinite crawlable combinations.
One retailer we audited had 4,200 products and 890,000 crawlable URLs, because four filter dimensions combined freely in the URL. Googlebot was spending most of its time on colour-and-size combinations that returned zero results.
4. Server logs show crawlers stuck in low-value directories
Log file analysis is the only way to see what crawlers actually do rather than what you assume. Export a month of logs, filter to verified Googlebot, and group requests by directory. If a large share of hits land on parameter URLs, tag archives or pagination beyond page five, that is budget not being spent on your commercial pages.
5. Redirect chains everywhere
Every hop in a redirect chain is another request. Sites that have been through two or three migrations frequently accumulate chains three and four hops deep. Flattening them so every redirect points directly at the final destination is unglamorous work that reliably frees up meaningful crawl capacity.
What to do about it
- Block genuinely useless parameter combinations in robots.txt, and make sure you are not blocking resources needed to render pages.
- Apply canonical tags consistently, and confirm your internal links and sitemap agree with them.
- Flatten every redirect chain to a single hop.
- Remove dead URLs from your XML sitemaps — a sitemap full of 404s teaches Google to trust it less.
- Improve server response time. Faster responses mean more URLs fetched per visit.
Crawl budget problems are rarely the only thing holding a site back, but they are frequently the reason other fixes appear not to work. If Google cannot reach the page, nothing you do on the page matters.