Skip to main content

​
Site Readiness

Before an AI engine can cite you, its crawler has to be allowed to read your site. Site Readiness is the access layer of GEO: it tells you who is allowed in, and gives you the files that tell AI systems what to read once they are. Navigate to Optimize > Site Readiness in the left navigation menu.

​
AI Crawler Readiness

Site Readiness checks your robots.txt against 63 known AI crawlers — from the engines behind ChatGPT, Claude, Gemini, Perplexity, Copilot, and Mistral, down to the long tail of smaller bots. For each crawler, the view shows:
  • Allowed or blocked — the verdict for that specific bot against your current robots.txt.
  • The exact rule that applies — so you know precisely which Disallow/Allow line is responsible, instead of guessing.
  • What the crawler is for — a short description of the product or engine behind it.
  • Priority tier — crawlers are grouped so the engines that matter most for your visibility are front and center, rather than buried in a long alphabetical list.
Each crawler entry also shows a last seen time, pulled from your connected server logs. Readiness tells you who is allowed in; last seen tells you who actually came. Cross-reference with Crawler Analytics to see the full picture — a crawler can be allowed but never show up, which usually points to a discovery problem rather than an access problem.

​
robots.txt analysis

Site Readiness checks your existing robots.txt file and provides an AI-optimized version to ensure all major AI crawlers can access and understand your content structure. If a rule is unintentionally blocking a crawler you want to allow, the diagnostic flags it directly.

​
llms.txt and llms-full.txt

  • llms.txt — a lightweight configuration file that helps AI models understand your site’s structure and key content. It points AI systems at your best content instead of leaving them to guess.
  • llms-full.txt — a more detailed version of llms.txt, providing comprehensive content for AI models to understand your business in depth.
Both are generated automatically based on your current site setup.

​
How do I generate or update these files?

Click the Generate button for each file. Qwairy auto-generates the content based on your existing site setup. You can then preview and download the files, and re-generate them any time your site structure changes.