Skip to main content

​
Overview

Crawler Analytics tracks AI bot visits to your website from platforms like OpenAI, Anthropic, Google, and Perplexity.

​
KPIs

MetricDescription
Total CrawlsAI crawler visits in the selected period
Unique CrawlersDistinct AI bots that visited
Top CrawlerBot with the most visits (with logo)
Avg Daily CrawlsAverage visits per day
Each KPI shows a trend indicator comparing to the previous period.

​
Tracked Crawlers

24 crawlers from major AI providers:
ProviderCrawlers
OpenAIGPTBot, ChatGPT-User, OAI-SearchBot
AnthropicClaudeBot, Claude-User, Claude-SearchBot, Claude-Web
GoogleGoogle-Extended
MicrosoftBingBot
PerplexityPerplexityBot, Perplexity-User
AppleApplebot, Applebot-Extended
AmazonAmazonbot
MetaMeta-ExternalAgent, Meta-ExternalFetcher, FacebookExternalHit
ByteDanceBytespider
You.comYouBot
Common CrawlCCBot
OthersDiffbot, Cohere-AI, Panscient

​
Crawler Types

Live Search - Fetch content for user queries:
  • ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot, Claude-SearchBot
Training - Index content for model training:
  • GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, Cohere-AI, Panscient
Platform - Other platform crawlers

​
Charts

​
Crawler Type Distribution

Pie chart showing breakdown by purpose (Live Search vs Training vs Platform) with percentage and visit count for each type.

​
Crawler Distribution

Pie chart showing visits by individual bot with logo, percentage, and trend. Shows top 6 crawlers with “View all X crawlers” dialog for complete list. Stacked bar chart showing daily activity by crawler over time.

​
Top Crawled Pages

Table showing:
ColumnDescription
PageURL path with external link
VisitsTotal crawler visits
CrawlersBreakdown by bot (logos + count)
UniqueNumber of distinct crawlers
Status CodesHTTP response distribution (200, 404, etc.)
TypesLive Search / Training / Other counts
Sortable by all columns. Paginated (10 per page).

​
Integrations

IntegrationTierMethod
VercelProLog Drain
Cloudflare WorkersFreeEdge Worker
Cloudflare LogpushEnterpriseLogpush
NetlifyProLog Drains
FastlyProHTTPS Logging
AkamaiEnterpriseDataStream 2
Generic HTTP APIFreeHTTP API
AWS CloudFrontEnterpriseComing soon

​
Setup

​
Vercel

  1. In Vercel: Settings > Log Drains > Add Log Drain
  2. Configure:
    • Sources: Static and Function
    • Environment: Production
    • Format: JSON
    • Sampling: 100%
  3. Endpoint: https://www.qwairy.co/api/logs/vercel
  4. Header: x-qwairy-token with your token
  5. Verify and enable in Qwairy

​
Cloudflare Workers

  1. Go to Analytics > Crawler Analytics > Setup
  2. Select Cloudflare Workers
  3. Create API key
  4. Copy the Worker script
  5. Deploy in Cloudflare dashboard

​
Generic HTTP API

Send POST requests with: 
{
  "status_code": 200,
  "request_method": "GET",
  "request_path": "/blog/article",
  "user_agent": "GPTBot/1.0",
  "timestamp": "2024-01-15T10:30:00Z"
}

​
Date Ranges

RangeDescription
Last 24 hoursRecent activity
Last 7 daysWeekly view
Last 30 daysDefault
Last 60 daysExtended
Last 90 daysQuarterly
All time (1 year)Admin only

​
Settings

Click the Settings button (visible when configured) to access integration configuration.