TL;DR — Open Analytics > Crawler Analytics to see which of the 24 tracked AI bots are visiting your site, how often, and which pages they hit. Cross-check with Analytics > Page Performance to verify your highest-value URLs are being crawled by each major bot within the last 30 days. If a primary crawler is missing entirely, head to Technical > Technical Analysis to diagnose the block. Pro tip: filter Page Performance to your conversion-critical pages first — if GPTBot has never visited your pricing page, that page is invisible to ChatGPT.
The Question
“Which AI crawlers are visiting my site and how frequently?”Every major AI model is backed by a web crawler that gathers training and retrieval data. Knowing which bots visit your site — and how often — tells you which AI products are likely to have your content in their index. A crawler that never visits means an AI model that never learned from you. This page shows you where to find that data in Qwairy and how to act on it. You might also be wondering:
- “Is GPTBot allowed to crawl my site, or am I accidentally blocking it?”
- “Which pages do AI bots visit most, and which ones do they ignore?”
- “How does my crawler traffic compare month over month?”
Where to Go in Qwairy
Start here: Analytics > Crawler Analytics
Navigate to Analytics > Crawler Analytics — your primary view for bot traffic.
The top panel shows a ranked list of AI crawlers with total visit counts, unique pages crawled, and last-seen timestamps for the selected period.
Qwairy tracks 24 AI bots including GPTBot (ChatGPT), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini), Bingbot/OAI, Applebot-Extended, and more.
Go deeper: Analytics > Page Performance
Cross-reference with Analytics > Page Performance.
This view shows which individual URLs received crawler visits from each bot, letting you identify which strategic pages are in each AI model’s crawl queue and which are being skipped.
Use the Crawler filter to isolate a specific bot and sort by visit count to see its crawl priorities on your site.
Complete the picture: Technical > Technical Analysis
Connect findings to Technical > Technical Analysis to understand why certain bots are absent or why specific pages are not being crawled.
The robots.txt and llms.txt audit in Technical Analysis directly explains crawl gaps visible in Crawler Analytics — the two views are designed to be read together.
What to Look For
Crawler Analytics — Bot Visit Overview
The main table in Crawler Analytics is the authoritative record of which AI systems are actively ingesting your content. Frequent visits from a crawler strongly correlate with your content appearing in that AI model’s responses.| Element | What it tells you |
|---|---|
| Visit count per bot | Raw frequency of crawl activity — higher means more up-to-date data in that AI’s index |
| Unique pages crawled | Breadth of coverage — a bot hitting 3 pages out of 200 is a depth problem |
| Last seen | Recency of the most recent crawl — stale timestamps suggest your domain is deprioritized |
| HTTP status breakdown | Whether bots received 200 OK, 301 redirects, 403 blocks, or 404s |
| Crawl frequency trend | Week-over-week or month-over-month change in visit volume |
Page Performance — Per-URL Crawler Breakdown
The per-page view reveals crawl depth gaps. A bot may visit your homepage regularly but never reach your product or pricing pages — those pages are effectively invisible to that AI model even though the crawler technically has access.Pro Tip: Filter Page Performance to your highest-converting pages and verify each major AI crawler has visited within the last 30 days. If any are missing, those pages are likely absent from that AI model’s retrieval index.
Filters That Help
| Filter | How to use it for this question |
|---|---|
| Bot / Crawler | Isolate a single user-agent to audit its specific behavior and reach on your site |
| Period | Compare 30d vs 90d to identify whether crawl frequency is stable, increasing, or declining after a site change |
| Page / URL section | Filter to a specific subdirectory (e.g., /blog/, /pricing/) to check whether that section is being crawled by AI bots |
How to Interpret the Results
Good result
All four primary AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — show active visits within the last 14 days, with at least 40% of your key pages crawled by each. HTTP status codes are predominantly 200 OK with no 403 or blocked patterns. Crawl frequency is stable or growing.Needs attention
One or more primary crawlers showing zero visits over a 30-day period, or a crawler that visits only 1-3 pages while the rest of your site returns no bot traffic. A pattern of 403 status codes points to a firewall or robots.txt block. A last-seen timestamp older than 60 days on a key crawler means your content on that AI platform may be significantly out of date.Example
Scenario: Your online news publication is frequently cited by Perplexity when readers ask about breaking stories, but your articles almost never appear in ChatGPT responses. You want to determine if this is a crawl access issue or a content authority problem.
- Open Analytics > Crawler Analytics and compare PerplexityBot vs GPTBot visit counts over the last 90 days. PerplexityBot shows 1,240 visits across 310 article pages. GPTBot shows 18 visits across 5 pages — all on your homepage, about page, and editorial policy page.
- Switch to Analytics > Page Performance and filter to GPTBot. Your investigative reports, opinion columns, and breaking news archives have zero GPTBot visits. The HTTP status column shows 200 OK for those pages — the crawler can access them, it simply has not discovered them.
- Navigate to Technical > Technical Analysis to check your sitemap and llms.txt configuration. The audit flags that your XML sitemap is not declared in robots.txt and that your news sitemap uses a non-standard URL pattern that GPTBot does not follow. Add the sitemap directive and create an llms.txt highlighting your editorial sections. Monitor GPTBot coverage over the following 30 days in Crawler Analytics to confirm it begins indexing your article pages.
Go Further
Crawler visit tracking dashboard
Build a crawler visit tracking dashboard in Looker Studio to monitor which AI bots are crawling your site
Crawler Analytics feature guide
Read the Crawler Analytics documentation to understand how Qwairy detects and reports AI crawler activity
robots.txt and llms.txt guide
Read the complete guide to configuring robots.txt and llms.txt for controlling AI crawler access
Related Questions
Is my website properly optimized for AI crawlers?
Run the full AI readiness audit to fix the issues behind your crawl gaps
Do I have the right robots.txt and llms.txt configuration?
Verify your crawler directives are allowing the bots you want in and blocking the ones you don’t
Which pages are crawled but never cited?
Go beyond crawl data to find the pages AI bots visit but never reference in responses
How visible is my brand in AI-powered search engines?
Connect crawler access to actual brand visibility in AI-generated answers