Skip to main content

Direct answer

Open Measure > Crawler Analytics to review retained crawler occurrences by page, state, and period, and inspect the supported crawler identities shown in the results. Report occurrence counts and retained page identities separately. These aggregates show activity accepted from the connected data source; they do not show which AI products ingested or used the content.

Data required

Use the classified User-Agent identity, normalized or pseudonymized page path, daily buckets, firstSeenAt, lastSeenAt, lifetime count, selected-period occurrences, status-code groups, production hostname, and coverage state. Keep provider and model names separate from crawler classifications.

Workflow

1

Validate collection

Confirm the monitored hostname, delivery coverage, analytic timezone, count precision, and classification rules before comparing crawlers.
2

Review user agents

Identify supported observable User-Agent identities and their occurrence counts in the selected period.
3

Inspect URL distribution

Compare retained page identities, selected-period occurrence counts, status-code groups, and site sections. The table shows the last crawl. For a page’s first retained HTTP 2xx observation and lifetime count, export Crawler Analytics and consult First seen at and Lifetime requests. Do not reconstruct pseudonymized paths.
4

Compare periods

Use equivalent windows and note site, logging, or classification changes that could affect the totals.

Interpretation

Occurrence count measures retained classified requests; page identities measure observed 2xx path coverage. Neither indicates indexing, training, retrieval, citation, or user traffic. Qwairy matches User-Agent tokens but does not verify source IP or reverse DNS.

Possible next actions

  • If a crawler has repeated error-group occurrences, test an access or infrastructure correction and compare a later daily rollup.
  • If only a narrow site section is visited, test discovery through valid sitemaps and internal links.
  • If activity changes abruptly, investigate logging, classification, deployment, and policy changes before attributing it externally.

Limitations

User-Agent identities can be missing, spoofed, renamed, or unsupported. Daily page rollups retain 30 analytic days; durable first-seen, last-seen, and lifetime counters are not a raw request history. Connected delivery can be partial, so no universal occurrence cadence defines healthy performance.

Source URLs in Looker Studio

Review recorded citation URLs separately from crawler occurrences.

Crawler Analytics

Review crawler data in the product.

Crawler configuration guide

Read additional configuration guidance.

Site optimization

Audit technical accessibility.

robots.txt and llms.txt

Review crawler directives.

Crawled but not cited

Compare crawler and citation observations.

Brand visibility

Analyze answer visibility separately.