Skip to main content

​
Direct answer

Use Optimize > Site Readiness to review public technical signals, then verify findings against your production site and Measure > Crawler Analytics. Resolve issues that conflict with your intended access policy. Passing an audit does not guarantee crawling, indexing, use, or citation.

​
Data required

Use the production hostname, representative public URLs, robots.txt and optional llms.txt, sitemap and canonical information, response behavior, Site Readiness findings, and retained Crawler Analytics rollups with their coverage state.

​
Workflow

1

Set the intended scope

List the public sections that relevant crawlers may access and the sections that must remain protected.
2

Review Site Readiness

Inspect findings for configuration, accessibility, discovery, and page-level diagnostics. Verify each result on the current production URL.
3

Check observed crawler activity

In Crawler Analytics, review supported identities, normalized or pseudonymized paths, first and last observed times, occurrence counts, status-code groups, and coverage warnings.
4

Prioritize verified issues

Address access errors, conflicting directives, broken discovery paths, or duplicate URL handling based on impact and change risk.

​
Interpretation

Technical readiness means that the selected public pages appear accessible and discoverable under the checks performed. It does not show that a provider indexed or used the content. Crawler occurrence rollups and recorded citations are independent observations.

​
Possible next actions

  • If important pages return errors to observed crawlers, test the smallest infrastructure or policy correction.
  • If crawling is concentrated on duplicate URLs, test canonical and internal-link normalization.
  • If technical access is healthy but citations are absent, investigate prompt relevance and content evidence separately.

​
Limitations

Audits are point-in-time checks and cannot represent every crawler implementation. JavaScript execution, regional infrastructure, authentication, rate limits, and third-party behavior can affect access. No technical configuration guarantees an answer mention or citation.

Site Readiness

Review the product audit and findings.

Crawler configuration guide

Read additional configuration guidance.

Qwairy videos

Browse product education videos.

Crawler occurrences

Inspect observed crawler activity.

robots.txt and llms.txt

Audit published crawler directives.

Crawled but not cited

Compare technical and citation observations.