SEO BasicsRankingCrawlingBasicsBlogStart Reading ->
<- Blog.View Site as Google

How to Read the Crawl Stats Report in Google Search Console

A complete technical guide to the Crawl Stats report in Google Search Console: understanding total requests, host status errors, and latency triage.

Oct 21, 2026.12 min read
Written by
Updated on: Oct 21, 2026Updated by: Muhammad Baig
How to Read the Crawl Stats Report in Google Search Console

Understanding how search engines discover and process your web pages is vital to maintaining search visibility. While third-party SEO crawlers can simulate a site crawl, they only estimate how search engines interact with your content. To see real, empirical data on how Googlebot crawls your server, you must inspect the Crawl Stats report.

Located inside Google Search Console, the Crawl Stats report provides a 90-day historical log of Googlebot requests across your entire domain. It reveals how often Google crawls your pages, how much bandwidth it consumes, how fast your server responds, and whether crawler connection errors threaten your search rankings.

According to official Google Search Console Crawl Stats documentation[1]Source 1Google Search Console Help. Crawl Stats Report: Detailed Data Overview and Host Status.View source ↗, this report is essential for detecting infrastructure bottlenecks, server downtime, and indexing delays before they damage organic traffic. Furthermore, as detailed in Google documentation on managing crawl budget[2]Source 2Google Search Central. Advanced SEO: Large Site Crawl Budget Management.View source ↗, poor server performance directly restricts Googlebot crawling capacity, causing fresh content to take days or weeks to index.

This guide provides a comprehensive technical walk-through of the Crawl Stats report. You will learn how to navigate to the report, how to interpret top-level summary metrics, how to evaluate the critical Host Status card, how to analyze crawl breakdown dimensions, and how to triage common crawling anomalies.

Where to Find and Access the Crawl Stats Report

The Crawl Stats report is located within the administrative settings of your Google Search Console account.

To access the report:

  1. Log into Google Search Console and select your domain property from the top-left dropdown menu.
  2. In the left-hand navigation menu, scroll down to the Settings section near the bottom.
  3. Under the Crawling category, locate the row labeled Crawl stats.
  4. Click Open Report to launch the comprehensive dashboard.

The report displays data covering the preceding 90 days. For website owners managing multiple subdomains, accessing the report via a Domain Property provides aggregated crawling statistics across all protocols (HTTP and HTTPS) and subdomains.

The Top-Level Summary Dashboard

Technical architecture diagram detailing the structure of Google Search Console's Crawl Stats report across summary metrics, host status, and breakdown dimensions.
Analyzing the Crawl Stats dashboard requires evaluating top-level performance metrics alongside multi-dimensional request breakdowns.

At the top of the report, Google presents three core time-series charts that summarize overall crawl health and server workload.

1. Total Crawl Requests

This metric represents the aggregate number of HTTP requests Googlebot made to your server on each day. A request is recorded every time Googlebot requests any resource, including HTML web pages, images, CSS files, JavaScript bundles, PDF downloads, or XML sitemaps.

Healthy sites typically display a consistent cadence of peaks and valleys corresponding to publishing schedules, inventory updates, and sitemap refreshes. Sharp spikes or severe collapses warrant immediate investigation.

2. Total Download Size

This chart measures the total volume of data in bytes that Googlebot downloaded from your server over the selected timeframe.

Comparing download size against total crawl requests reveals page weight efficiency. If your download size climbs dramatically while total crawl requests remain flat, your server may be delivering uncompressed assets, bloated JavaScript bundles, or oversized image formats that waste crawler bandwidth.

3. Average Response Time (Milliseconds)

Average response time measures the duration in milliseconds between Googlebot initiating an HTTP request and your web server delivering the full response headers and content payload.

In modern technical SEO, server latency directly controls crawl speed. When average response times remain below 300 to 500 milliseconds, Googlebot considers your server responsive and crawls freely. If response times climb above 1,000 to 2,000 milliseconds, Googlebot automatically reduces its crawl rate to prevent crashing your hosting infrastructure.

Host Status: The Infrastructure Health Card

Directly beneath the summary charts, Google Search Console displays the Host status evaluation. This section functions as an infrastructure health check, assessing whether your server reliably accepted connections over the past 90 days.

Google evaluates host availability across three technical pillars:

                            [ Host Status Evaluation Pillars ]
                                            |
         +----------------------------------+----------------------------------+
         |                                  |                                  |
         v                                  v                                  v
  [ 1. Robots.txt Fetch ]         [ 2. DNS Resolution ]         [ 3. Server Connectivity ]
  - Checks if /robots.txt         - Checks if domain name       - Checks if web server
    can be retrieved                resolves to server IP         returns 5xx errors or
  - 5xx error or timeout          - NXDOMAIN or DNS failure       drops TCP/TLS requests
    HALTS ALL CRAWLING              over 5% triggers error      - Timeouts cause backoff

1. Robots.txt Fetch

Before Googlebot crawls any URL on your website, it must verify access permissions by requesting /robots.txt. If your server returns a 5xx server error or connection timeout when Googlebot requests your robots file, Googlebot immediately pauses all crawling across your entire website.

Because Google cannot verify which paths are disallowed, it halts crawling to protect your private directories. If a robots.txt fetch error persists for several days, Google will begin suppressing existing indexed pages from search results. Ensuring that /robots.txt returns a clean 200 OK or 404 Not Found response is the most critical infrastructure requirement for search indexing.

2. DNS Resolution

DNS resolution evaluates whether your domain name servers successfully translated your domain name into an IP address when Googlebot attempted connection.

If your DNS provider experiences downtime, high packet loss, or misconfigured nameservers, Googlebot cannot locate your server. Google triggers a host availability warning if your DNS failure rate exceeds 5 percent on any given day.

3. Server Connectivity

Server connectivity assesses whether your web server accepted TCP connections and completed TLS handshakes. Connection timeouts, dropped packets, and HTTP 5xx server errors indicate that your hosting environment is overwhelmed, crash-prone, or aggressively blocking crawlers via misconfigured web application firewalls (WAF).

The Dotted Red Limit Line

Inside the Host status drilldown charts, Google renders a horizontal dotted red threshold line. If your error rate stays below this line, your host status remains marked as Good. If your connection error rate crosses above the red threshold, Google flags the status as Needs attention or Failing, and restricts crawl capacity until stability is restored.

Normal Fluctuations vs. Critical Host Problems

Not every change in your crawl charts indicates an emergency. Use the reference table below to distinguish between standard crawler behavior and genuine infrastructure failures:

Metric or Chart Pattern Typical Healthy Behavior Warning Signal Action Required
Total Crawl Requests Regular periodic oscillations following content releases and sitemaps Sudden 80% collapse in requests lasting multiple days Check Host Status for DNS or server connectivity errors
Total Crawl Requests Moderate temporary spike after publishing a new category or product line 500% sustained spike with no new content published Audit server logs for infinite URL parameter traps
Average Response Time Stable baseline fluctuating between 150 ms and 400 ms Sharp spike exceeding 1,500 ms lasting several consecutive days Optimize database queries, enable CDN caching, upgrade hosting
Robots.txt Fetch Continuous 100% success rate with zero connection failures Any failure spike crossing the dotted red threshold line Ensure /robots.txt returns 200 OK directly; check WAF rules
DNS Resolution 0% error rate across all Googlebot connection attempts Error rate crossing the 5% red threshold line Review nameserver configuration with your DNS provider
Server Connectivity Isolated momentary dips during scheduled off-peak maintenance Sustained connection timeouts and 502/503 response codes Inspect web server error logs; resolve backend resource exhaustion

Analyzing the Four Breakdown Dimensions

The bottom section of the Crawl Stats report groups Googlebot activity across four diagnostic dimensions. Analyzing these breakdowns helps you pinpoint where crawling resources are being spent or wasted.

1. Crawl Requests by Response

This dimension reveals the exact HTTP status codes Googlebot received across all crawl requests:

  • OK (200): The percentage of requests that returned content successfully. On healthy websites, 200 responses should account for the vast majority of all crawl activity (typically 75% to 90% or higher).
  • Moved permanently (301) & Temporarily (302): The percentage of requests that triggered redirects. While redirects are normal during site migrations, an excessively high redirect percentage indicates internal links pointing to outdated URLs rather than resolving destinations.
  • Not found (404): Requests hitting missing pages. A high 404 percentage suggests that your site has broken internal links or that external websites are linking to deleted URLs.
  • Server error (5xx): Requests resulting in backend application crashes or gateway timeouts. Any significant percentage of 5xx errors indicates infrastructure instability.
  • Blocked by robots.txt: Requests that Googlebot attempted to fetch but were prevented by robots.txt rules. As explained in our guide on robots.txt wildcards, reviewing blocked paths ensures you are not disallowing valuable content.

2. Crawl Requests by File Type

This dimension categorizes crawler requests based on resource format:

  • HTML: Web pages containing your primary content and internal linking structure.
  • Image: Photos, icons, and illustrations requested by general Googlebot or Googlebot-Image.
  • JavaScript & CSS: Front-end assets fetched to render the DOM. Under modern web rendering, Googlebot must download CSS and JavaScript files to verify mobile usability and page layout.
  • JSON: API endpoints, structured data feeds, or AJAX requests.
  • PDF & Document Formats: Standalone downloads requested by crawlers.

If JavaScript and CSS requests significantly outnumber HTML requests, your website may rely on heavily fragmented asset files. Consolidating CSS and JS bundles reduces crawler overhead.

3. Crawl Requests by Purpose

Google categorizes requests into two distinct operational objectives:

  • Discovery: Googlebot is requesting a URL that it has never encountered before. Discovery crawls occur when you launch new pages, submit new sitemaps, or when external sites link to new URLs.
  • Refresh: Googlebot is re-crawling an existing URL already present in its database to check for updated content, modified markup, or canonical changes.

On established websites, Refresh typically accounts for 80% to 90% of all crawl activity. If Discovery suddenly surges to 60% or higher without a major site release, your website may be generating infinite URL parameter loops through faceted navigation or calendar widgets.

4. Crawl Requests by Googlebot Type

This dimension shows which specialized Googlebot crawlers are visiting your website:

  • Googlebot Smartphone: The primary crawler for virtually all modern websites under Mobile-First Indexing. Smartphone requests should account for the majority of your HTML crawl activity.
  • Googlebot Desktop: Secondary crawler used to verify desktop rendering and parity.
  • Googlebot Image / Video: Specialized bots indexing media assets.
  • AdsBot & StoreBot: Commercial crawlers verifying landing page compliance and merchant center inventory.

Investigating Crawling Anomalies: Diagnostic Matrix

Diagnostic decision tree flowchart illustrating how to troubleshoot crawl request spikes, drops, high server latency, and host availability errors.
Use this diagnostic framework to identify whether crawl fluctuations stem from normal content updates or critical server bottlenecks.

When reviewing your Crawl Stats report, you will occasionally encounter sharp anomalies. Use the structured diagnostic matrix below to isolate root causes and apply corrective actions:

Observed Graph Anomaly Primary Suspect Cause Supporting Search Console Metric Corrective Technical Action
Sudden massive spike in crawl requests Infinite URL parameter generation or hacked site injection Surge in "Discovery" purpose and query strings Deploy robots.txt parameter disallow rules; clean up faceted filters
Sudden collapse in total crawl requests Host status failure, robots.txt fetch error, or site-wide block Collapse in 200 OK responses; rise in Host Status errors Verify /robots.txt access; check CDN and firewall IP blocks
Average response time climbs above 1,500 ms Uncached database queries, slow backend scripts, server load Flat or declining crawl requests; spike in 504 gateway timeouts Implement full-page caching; optimize slow database queries
High percentage of 301/302 responses Outdated internal linking pointing to redirected URL paths Surge in "Moved permanently" in response breakdown Update internal links and XML sitemaps directly to final 200 OK URLs
High percentage of 404 responses Deleted inventory or broken internal site navigation Check "Not found (404)" under Page Indexing report[4]Source 4Google Search Console Help. Page Indexing Report: Analyzing Crawled and Excluded Pages.View source ↗ Remove broken links from navigation; restore accidentally deleted pages

Step-by-Step Triage: Investigating a Crawl Latency Spike

If your Average Response Time chart begins climbing, take the following investigative steps:

                  [ Response Time Climbs Above 800 ms ]
                                     |
               Is the latency spike isolated to specific URLs
               or affecting the entire hosting infrastructure?
                                     |
                    +----------------+----------------+
                    |                                 |
           [ Isolated to Specific URLs ]     [ Site-Wide Latency Spike ]
                    |                                 |
         Check URL breakdown in Crawl Stats  Inspect server CPU/Memory usage
         and filter by Response Time         and database query slow logs
                    |                                 |
                    v                                 v
         Are the slow URLs dynamic,          Is the web server experiencing
         uncached search or filter pages?    traffic congestion or DDoS hits?
                    |                                 |
            +-------+-------+                 +-------+-------+
            |               |                 |               |
         [ YES ]          [ NO ]           [ YES ]          [ NO ]
            |               |                 |               |
     Add server caching  Optimize asset   Enable CDN edge  Upgrade hosting
     or disallow filter  sizes and heavy  caching & WAF    plan or database
     in robots.txt       DOM elements     rate limits      tier capacity
  1. Check Server Error Logs: Log into your web server and review Apache, Nginx, or LiteSpeed access and error logs matching the timestamps of the latency spike.
  2. Review Database Slow Query Logs: Identify unindexed SQL queries that lock database tables during concurrent crawler visits.
  3. Verify Googlebot Authenticity: Ensure that requests claiming to be Googlebot are legitimate crawlers rather than scrapers spoofing the user-agent string. Follow official Google Search Central guidance on verifying Googlebot[3]Source 3Google Search Central. Verifying Googlebot and Search Engine Crawlers.View source ↗ using reverse DNS lookups (host <ip> followed by host <domain>).
  4. Implement HTTP 304 Not Modified Caching: Ensure your server returns an HTTP 304 Not Modified header when Googlebot sends an If-Modified-Since header on unchanged pages. This eliminates the need to re-transmit the full HTML payload, slashing response times and download sizes.

Actionable Monthly Crawl Audit Checklist

Incorporate this five-step audit into your routine technical SEO maintenance:

  • Verify Host Status is Good: Ensure Robots.txt fetch, DNS resolution, and Server connectivity all display zero critical errors.
  • Monitor Average Response Time: Keep your server response time consistently below 500 ms to maximize Googlebot crawling efficiency.
  • Check Response Code Distributions: Confirm that 200 OK requests account for at least 80% of total crawl volume, and address any surging 404 or 5xx trends.
  • Evaluate Discovery vs. Refresh Balance: Ensure new pages are discovered promptly while core legacy pages receive regular refresh crawling.
  • Audit Asset Overhead: Review file type breakdowns to ensure CSS and JavaScript requests do not overwhelm HTML content indexing.

Sources

  1. Google Search Console Help. Crawl Stats Report: Detailed Data Overview and Host Status.

  2. Google Search Central. Advanced SEO: Large Site Crawl Budget Management.

  3. Google Search Central. Verifying Googlebot and Search Engine Crawlers.

  4. Google Search Console Help. Page Indexing Report: Analyzing Crawled and Excluded Pages.

Share

About the Contributors