What is Crawl Budget? How to Optimize for Large Sites

Crawl budget is generally not a concern for websites under 1,000 pages with good technical health. Google crawls small sites completely and regularly. Focus on crawl budget optimization only if you have 10,000+ pages, significant faceted navigation, or evidence in Search Console that important pages are not being indexed.

Crawling = Googlebot visits and reads your page content. Indexing = Google adds that page to its search database and makes it eligible to appear in search results. A page can be crawled but not indexed (due to quality issues, noindex tag, or thin content). Use URL Inspection in Search Console to check both stages for any specific URL.

Block with robots.txt Disallow or noindex: (1) Faceted navigation URLs (filter combinations on e-commerce: /products?color=red&size=large creates thousands of near-duplicate pages), (2) URL parameters that add tracking or session data but do not change content (?utm_source=, ?sessionid=), (3) Staging or development environments, (4) Duplicate content (printer-friendly pages, alternative sort orders with same products), (5) Low-value utility pages (login, cart, checkout for non-product content), (6) Admin or internal tools accessible to Googlebot. Do NOT block: pages you want indexed, pages with valuable unique content, pages that pass link equity to important pages.

URL parameters create multiple URLs for the same content. Example: /products/shoes can become /products/shoes?color=blue, /products/shoes?sort=price-asc, /products/shoes?page=2 — each a separate URL Googlebot may crawl. For a 10,000-product catalog with 5 filter types and 3 sort orders, this creates 150,000+ crawlable URLs for 10,000 actual products. Manage this: use canonical tags pointing all parameter variants to the canonical product/category URL, configure URL Parameter handling in Google Search Console (legacy feature), and block non-indexable parameter patterns in robots.txt.