Skip to main content

​
Direct answer

Open Measure > Crawler Analytics and filter retained occurrence rollups by supported User-Agent identity, page, state, and period. Report occurrence counts and retained page identities separately. These aggregates show activity accepted from the connected data source, not which AI products ingested or used the content.

​
Data required

Use the classified User-Agent identity, normalized or pseudonymized page path, daily buckets, firstSeenAt, lastSeenAt, lifetime count, selected-period occurrences, status-code groups, production hostname, and coverage state. Keep provider and model names separate from crawler classifications.

​
Workflow

1

Validate collection

Confirm the monitored hostname, delivery coverage, analytic timezone, count precision, and classification rules before comparing crawlers.
2

Review user agents

Identify supported observable User-Agent identities and their occurrence counts in the selected period.
3

Inspect URL distribution

Compare retained page identities, occurrence counts, status-code groups, and site sections. Do not reconstruct pseudonymized paths.
4

Compare periods

Use equivalent windows and note site, logging, or classification changes that could affect the totals.

​
Interpretation

Occurrence count measures retained classified requests; page identities measure observed 2xx path coverage. Neither indicates indexing, training, retrieval, citation, or user traffic. Qwairy matches User-Agent tokens but does not verify source IP or reverse DNS.

​
Possible next actions

  • If a crawler has repeated error-group occurrences, test an access or infrastructure correction and compare a later daily rollup.
  • If only a narrow site section is visited, test discovery through valid sitemaps and internal links.
  • If activity changes abruptly, investigate logging, classification, deployment, and policy changes before attributing it externally.

​
Limitations

User-Agent identities can be missing, spoofed, renamed, or unsupported. Daily page rollups retain 30 analytic days; durable first-seen, last-seen, and lifetime counters are not a raw request history. Connected delivery can be partial, so no universal occurrence cadence defines healthy performance.

Source URLs in Looker Studio

Report URL-level observations.

Crawler Analytics

Review crawler data in the product.

Crawler configuration guide

Read additional configuration guidance.

Site optimization

Audit technical accessibility.

robots.txt and llms.txt

Review crawler directives.

Crawled but not cited

Compare crawler and citation observations.

Brand visibility

Analyze answer visibility separately.