Documentation Index
Fetch the complete documentation index at: https://docs.qwairy.co/llms.txt
Use this file to discover all available pages before exploring further.
Overview
Crawler Analytics tracks AI bot visits to your website from platforms like OpenAI, Anthropic, Google, and Perplexity.
KPIs
| Metric | Description |
|---|
| Total Crawls | AI crawler visits in the selected period |
| Unique Crawlers | Distinct AI bots that visited |
| Top Crawler | Bot with the most visits (with logo) |
| Avg Daily Crawls | Average visits per day |
Tracked Crawlers
24 crawlers from major AI providers:
| Provider | Crawlers |
|---|
| OpenAI | GPTBot, ChatGPT-User, OAI-SearchBot |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web |
| Google | Google-Extended |
| Microsoft | BingBot |
| Perplexity | PerplexityBot, Perplexity-User |
| Apple | Applebot, Applebot-Extended |
| Amazon | Amazonbot |
| Meta | Meta-ExternalAgent, Meta-ExternalFetcher, FacebookExternalHit |
| ByteDance | Bytespider |
| You.com | YouBot |
| Common Crawl | CCBot |
| Others | Diffbot, Cohere-AI, Panscient |
Crawler Types
Live Search - Fetch content for user queries:
- ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot, Claude-SearchBot
Training - Index content for model training:
- GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, Cohere-AI, Panscient
Platform - Other platform crawlers
Charts
Crawler Type Distribution
Pie chart showing breakdown by purpose (Live Search vs Training vs Platform) with percentage and visit count for each type.
Crawler Distribution
Pie chart showing visits by individual bot with logo, percentage, and trend. Shows top 6 crawlers with “View all X crawlers” dialog for complete list.
Crawler Trends
Stacked bar chart showing daily activity by crawler over time.
Top Crawled Pages
Table showing:
| Column | Description |
|---|
| Page | URL path with external link |
| Visits | Total crawler visits |
| Crawlers | Breakdown by bot (logos + count) |
| Unique | Number of distinct crawlers |
| Status Codes | HTTP response distribution (200, 404, etc.) |
| Types | Live Search / Training / Other counts |
Integrations
| Integration | Tier | Method |
|---|
| Vercel | Pro | Log Drain |
| Cloudflare Workers | Free | Edge Worker |
| Cloudflare Logpush | Enterprise | Logpush |
| Netlify | Pro | Log Drains |
| Fastly | Pro | HTTPS Logging |
| Akamai | Enterprise | DataStream 2 |
| Generic HTTP API | Free | HTTP API |
| AWS CloudFront | Enterprise | Coming soon |
Setup
Vercel
- In Vercel: Settings > Log Drains > Add Log Drain
- Configure:
- Sources:
Static and Function
- Environment:
Production
- Format:
JSON
- Sampling:
100%
- Endpoint:
https://www.qwairy.co/api/logs/vercel
- Header:
x-qwairy-token with your token
- Verify and enable in Qwairy
Cloudflare Workers
- Go to Analytics > Crawler Analytics > Setup
- Select Cloudflare Workers
- Create API key
- Copy the Worker script
- Deploy in Cloudflare dashboard
Generic HTTP API
Send POST requests with:
{
"status_code": 200,
"request_method": "GET",
"request_path": "/blog/article",
"user_agent": "GPTBot/1.0",
"timestamp": "2024-01-15T10:30:00Z"
}
Date Ranges
| Range | Description |
|---|
| Last 24 hours | Recent activity |
| Last 7 days | Weekly view |
| Last 30 days | Default |
| Last 60 days | Extended |
| Last 90 days | Quarterly |
| All time (1 year) | Admin only |
Settings
Click the Settings button (visible when configured) to access integration configuration.
