Skip to main content

​
Direct answer

Compare the last 30 days of page rollups in Measure > Crawler Analytics with SELF URLs recorded in Monitor > Source Explorer for an aligned period. A page with crawler occurrences and no recorded citation is an investigation candidate, not proof that a provider evaluated and rejected it.

​
Data required

Use reconstructible normalized paths, supported crawler identity, daily buckets, first and last observed times, status-code groups, coverage state, stored answers, recorded citation URLs, providers, topics, and configured prompts. Exclude pseudonymized paths from URL matching and document redirects or variants.

​
Workflow

1

Align the datasets

Choose comparable periods and normalize protocol, hostname, trailing slash, parameters, redirects, and canonical variants.
2

Find crawler-only URLs

Identify reconstructible pages with selected-period crawler occurrences but no matching SELF citation URL in the selected stored answers.
3

Check technical context

Review response status, access configuration, canonicalization, internal discovery, and the Site Readiness findings for each candidate.
4

Review content relevance

Compare the page with the monitored prompts and sources recorded for those answers. Exclude pages that do not address the observed need.

​
Interpretation

A crawler occurrence shows a request retained in an aggregate for a supported User-Agent identity. It does not establish indexing, training, retrieval, or future citation. A citation is a source URL recorded with a stored answer; it does not require an observed occurrence in the connected period.

​
Possible next actions

  • If URL variants split the data, test canonicalization or reporting normalization before changing content.
  • If error-group occurrences recur, test a technical correction and compare later daily rollups.
  • If access is healthy but relevance is weak, test a content update against a defined prompt cohort.

​
Limitations

Crawler and citation datasets have different coverage, retention, and URL representations. Page rollups retain 30 analytic days, pseudonymized paths cannot be joined, and some provider activity does not expose a supported User-Agent identity. No occurrence frequency guarantees a citation.

Crawler Analytics

Review observed crawler activity.

Source URLs in Looker Studio

Compare URL-level data in a documented source.

Content Opportunities

Review observed content gaps.

Gap to article

Run a documented content experiment.

Site optimization

Audit technical accessibility.

Pages cited by AI

Analyze recorded citation URLs.