Skip to main content

​
Direct answer

Few Google requests in Measure > Crawler Analytics usually reflect how Google crawls, not low visibility in Google. Google Search, including AI Overviews and AI Mode, relies on Googlebot, the same crawler as traditional search. Crawler Analytics counts AI-specific crawlers only, so it excludes Googlebot: a Googlebot request does not show whether the page will serve a traditional result or an AI feature. The Google identities that Crawler Analytics can count fetch pages for narrower uses, often on a user’s request, so low volumes are expected. To measure whether your brand appears in Google’s AI answers, use answer monitoring rather than crawler counts.

​
Data required

Use Crawler Analytics occurrences by User-Agent identity for the selected period, with the connector’s coverage state. For Google’s AI features, use your monitored prompts and the answers recorded for the Google providers selected in your workspace. For Googlebot activity, use your server or CDN logs and the crawl reports in Google Search Console.

​
Workflow

1

Confirm collection

Check that the connector is connected, that the coverage state is complete for the period, and that no upstream filter removes Google User-Agents you expect to see.
2

Identify the Google rows

Compare the observed identities with the table in How Google crawlers map to Crawler Analytics. Only identities from the public set appear in totals and tables.
3

Measure visibility in Google's AI features

In Monitor > Prompt Tracking, review the answers recorded for Google providers. Mentions and citations show whether your brand appears in those answers.
4

Review Googlebot crawling separately

Use Google Search Console crawl statistics or your raw server logs to review how Googlebot crawls the site. Treat this activity as search crawling, not as an AI-specific signal.

​
How Google crawlers map to Crawler Analytics

Google documents these identities. The column on the right describes what Crawler Analytics does with each one. Sources: AI features and your website (updated 10 December 2025), Google’s common crawlers (updated 15 October 2026), and Google’s user-triggered fetchers (updated 19 August 2026). Verified on 30 October 2026.

​
Interpretation

A low Google count in Crawler Analytics does not measure your visibility in AI Overviews, AI Mode, or Gemini. Those features can use content that Googlebot already crawled for Search, so an AI answer does not require a new, identifiable request in your logs. Google rows that do appear describe specific uses. Google-GeminiNotebook shows that users added your pages as sources in Gemini Notebook. Google-CloudVertexBot shows crawls requested for Vertex AI agents. Neither represents Google Search as a whole.

​
Possible next actions

  • To track whether your brand appears in Google’s AI answers, monitor prompts with the Google providers available in your workspace and compare mention and citation rates over time.
  • To check that Google can crawl important pages, review Search Console crawl statistics and the status codes Googlebot receives in your server logs.
  • To change how Google may use your content for Gemini training and grounding, review the Google-Extended rules in your robots.txt with Optimize > Site Readiness.

​
Limitations

Crawler Analytics classifies the User-Agent string it receives and does not verify the requester’s IP address or reverse DNS. A User-Agent can be spoofed. Google can rename or add identities. This page describes Google documentation on the verification date above; recheck the linked sources before acting on a specific identity.

Crawler Analytics

Read crawler data and the public identity set.

Which AI crawlers visit my site?

Review crawler occurrences by identity and page.

robots.txt and llms.txt

Review crawler directives.

Crawled but not cited

Compare crawler and citation observations.