Skip to main content

​
Overview

Crawler Analytics tracks AI bot visits to your website from platforms like OpenAI, Anthropic, Google, and Perplexity.

​
KPIs

Each KPI shows a trend indicator comparing to the previous period.

​
Tracked Crawlers

Crawlers from major AI providers:

​
Crawler Types

Live Search - Fetch content for user queries:
  • ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot, Claude-SearchBot
Training - Index content for model training:
  • GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, Cohere-AI, Panscient
Platform - Other platform crawlers

​
Charts

​
Crawler Type Distribution

Pie chart showing breakdown by purpose (Live Search vs Training vs Platform) with percentage and visit count for each type.

​
Crawler Distribution

Pie chart showing visits by individual bot with logo, percentage, and trend. Shows top 6 crawlers with “View all X crawlers” dialog for complete list. Stacked bar chart showing daily activity by crawler over time.

​
Top Crawled Pages

Table showing: Sortable by all columns. Paginated (10 per page).

​
Integrations

​
Setup

​
Vercel

  1. In Vercel: Settings > Log Drains > Add Log Drain
  2. Configure:
    • Sources: Static and Function
    • Environment: Production
    • Format: JSON
    • Sampling: 100%
  3. Endpoint: https://www.qwairy.co/api/logs/vercel
  4. Header: x-qwairy-token with your token
  5. Verify and enable in Qwairy

​
Cloudflare Workers

  1. Go to Measure > Crawler Analytics > Setup
  2. Select Cloudflare Workers
  3. Create API key
  4. Copy the Worker script
  5. Deploy in Cloudflare dashboard

​
Generic HTTP API

Send POST requests with:

​
Date Ranges

​
Settings

Click the Settings button (visible when configured) to access integration configuration.

​
Also available via MCP

This data is also queryable through the Qwairy MCP server — the get_crawler_activity tool returns the same figures for use in Claude, Cursor, or any MCP-compatible assistant. Requires the read:measure scope.