Crawler and robots-policy access
The access table checks configured crawler and robots-policy identities against the site’s current robots.txt rules. Use it to identify whether each identity appears allowed, blocked, mixed by path, or could not be evaluated. Some entries are observable crawler User-Agents, such as GPTBot. Others are policy tokens, such asGoogle-Extended, that control permitted uses of content fetched by another crawler. A policy-token result cannot produce a Crawler Analytics occurrence or last-seen timestamp.
The result is a robots-policy observation. It does not show that a provider has crawled, indexed, cited, or will use the site. Only the supported observable User-Agent subset can appear in Crawler Analytics.
robots.txt
The robots.txt section reads the public file and summarizes relevant directives. Review the raw file before changing it, especially when rules differ by user agent or path. If Qwairy cannot retrieve the file, check the URL, status code, redirects, firewall, and authentication requirements.llms.txt
The llms.txt section helps you inspect or prepare a text file that describes selected site resources. Support and interpretation vary across AI systems. Publishing the file does not guarantee discovery or citation. If Qwairy generates suggested content, review every URL and description before placing it on the public site.Recommended review sequence
- Confirm the monitored domain and protocol.
- Inspect the current robots.txt response.
- Review user-agent-specific rules.
- Check the proposed llms.txt content, if used.
- Coordinate deployment with the site owner.
- Re-run the check after the public files change.