> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qwairy.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawler Analytics

> Monitor AI crawler activity on your website

## Overview

Crawler Analytics tracks AI bot visits to your website from platforms like OpenAI, Anthropic, Google, and Perplexity.

## KPIs

| Metric               | Description                              |
| -------------------- | ---------------------------------------- |
| **Total Crawls**     | AI crawler visits in the selected period |
| **Unique Crawlers**  | Distinct AI bots that visited            |
| **Top Crawler**      | Bot with the most visits (with logo)     |
| **Avg Daily Crawls** | Average visits per day                   |

Each KPI shows a trend indicator comparing to the previous period.

## Tracked Crawlers

Crawlers from major AI providers:

| Provider         | Crawlers                                                      |
| ---------------- | ------------------------------------------------------------- |
| **OpenAI**       | GPTBot, ChatGPT-User, OAI-SearchBot                           |
| **Anthropic**    | ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web          |
| **Google**       | Google-Extended                                               |
| **Microsoft**    | BingBot                                                       |
| **Perplexity**   | PerplexityBot, Perplexity-User                                |
| **Apple**        | Applebot, Applebot-Extended                                   |
| **Amazon**       | Amazonbot                                                     |
| **Meta**         | Meta-ExternalAgent, Meta-ExternalFetcher, FacebookExternalHit |
| **ByteDance**    | Bytespider                                                    |
| **You.com**      | YouBot                                                        |
| **Common Crawl** | CCBot                                                         |
| **Others**       | Diffbot, Cohere-AI, Panscient                                 |

### Crawler Types

**Live Search** - Fetch content for user queries:

* ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot, Claude-SearchBot

**Training** - Index content for model training:

* GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, Cohere-AI, Panscient

**Platform** - Other platform crawlers

## Charts

### Crawler Type Distribution

Pie chart showing breakdown by purpose (Live Search vs Training vs Platform) with percentage and visit count for each type.

### Crawler Distribution

Pie chart showing visits by individual bot with logo, percentage, and trend. Shows top 6 crawlers with "View all X crawlers" dialog for complete list.

### Crawler Trends

Stacked bar chart showing daily activity by crawler over time.

## Top Crawled Pages

Table showing:

| Column           | Description                                 |
| ---------------- | ------------------------------------------- |
| **Page**         | URL path with external link                 |
| **Visits**       | Total crawler visits                        |
| **Crawlers**     | Breakdown by bot (logos + count)            |
| **Unique**       | Number of distinct crawlers                 |
| **Status Codes** | HTTP response distribution (200, 404, etc.) |
| **Types**        | Live Search / Training / Other counts       |

Sortable by all columns. Paginated (10 per page).

## Integrations

| Integration            | Tier       | Method        |
| ---------------------- | ---------- | ------------- |
| **Vercel**             | Pro        | Log Drain     |
| **Cloudflare Workers** | Free       | Edge Worker   |
| **Cloudflare Logpush** | Enterprise | Logpush       |
| **Netlify**            | Pro        | Log Drains    |
| **Fastly**             | Pro        | HTTPS Logging |
| **Akamai**             | Enterprise | DataStream 2  |
| **Generic HTTP API**   | Free       | HTTP API      |
| **AWS CloudFront**     | Enterprise | Coming soon   |

## Setup

### Vercel

1. In Vercel: **Settings > Log Drains > Add Log Drain**
2. Configure:
   * Sources: `Static` and `Function`
   * Environment: `Production`
   * Format: `JSON`
   * Sampling: `100%`
3. Endpoint: `https://www.qwairy.co/api/logs/vercel`
4. Header: `x-qwairy-token` with your token
5. Verify and enable in Qwairy

### Cloudflare Workers

1. Go to **Measure > Crawler Analytics > Setup**
2. Select **Cloudflare Workers**
3. Create API key
4. Copy the Worker script
5. Deploy in Cloudflare dashboard

### Generic HTTP API

Send POST requests with:

```json theme={null}
{
  "status_code": 200,
  "request_method": "GET",
  "request_path": "/blog/article",
  "user_agent": "GPTBot/1.0",
  "timestamp": "2024-01-15T10:30:00Z"
}
```

## Date Ranges

| Range             | Description     |
| ----------------- | --------------- |
| Last 24 hours     | Recent activity |
| Last 7 days       | Weekly view     |
| Last 30 days      | Default         |
| Last 60 days      | Extended        |
| Last 90 days      | Quarterly       |
| All time (1 year) | Admin only      |

## Settings

Click the **Settings** button (visible when configured) to access integration configuration.

## Also available via MCP

This data is also queryable through the Qwairy MCP server — the [`get_crawler_activity`](/mcp/tools/measure) tool returns the same figures for use in Claude, Cursor, or any MCP-compatible assistant. Requires the `read:measure` scope.

## Related use cases

* [Which AI crawlers are visiting my site and how frequently?](/use-cases/technical/which-ai-crawlers-visit)
* [Which pages are crawled by AI bots but never cited?](/use-cases/technical/pages-crawled-but-never-cited)