Skip to main content

​
Direct answer

Use Optimize > Site Readiness to inspect the files Qwairy can observe, then compare the result with your intended access policy and retained Crawler Analytics rollups. Validate robots.txt syntax and path scope. Treat llms.txt as an optional published file whose interpretation can vary by consumer.

​
Data required

Use the exact public robots.txt and llms.txt URLs, intended allow or disallow policy by path and identity, sitemap locations, production hostname, Site Readiness findings, and Crawler Analytics occurrence, status-group, and coverage aggregates.

​
Workflow

1

Define the policy

Decide which public paths should be accessible to each relevant crawler class. Keep private or sensitive content protected by authentication, not crawler directives alone.
2

Review the published files

Inspect robots.txt and llms.txt from the production hostname. Check syntax, path matching, host variants, and referenced URLs.
3

Run Site Readiness

Review reported configuration issues and verify each one against the actual files before changing production behavior.
4

Compare observed activity

Use Crawler Analytics to see whether supported observable User-Agent identities have occurrences on allowed paths and whether error-group occurrences repeat.

​
Interpretation

A syntactically valid directive expresses your policy but does not guarantee that every crawler will visit, index, use, or cite a page. llms.txt support and behavior can differ between consumers. Crawler rollups provide observations, not proof of ingestion or model training.

​
Possible next actions

  • If an intended public path is blocked, test the smallest policy correction after security and content-owner review.
  • If host or path variants conflict, test a normalized configuration and verify the published response.
  • If configuration is valid but no occurrences appear, investigate discovery and delivery coverage before assuming crawler rejection.

​
Limitations

Crawler policies and implementations can change outside Qwairy. robots.txt is not an access-control system, and llms.txt is not a guarantee of use. Configuration changes can expose paths unintentionally, so review production scope before publishing.

Crawler configuration guide

Read additional configuration guidance.

Site Readiness

Review the product audit.

Qwairy videos

Browse product education videos.

Site optimization

Review the broader technical workflow.

Crawler occurrences

Inspect observed user agents and URLs.

Crawled but not cited

Compare crawler and citation datasets.