> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qwairy.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloudflare Worker

> Deploy Qwairy's generated edge collector on a Cloudflare Worker route and verify crawler delivery safely.

Use the Cloudflare Worker collector when the monitored site is proxied through Cloudflare but Cloudflare Logpush is not the selected delivery source. The Worker filters known AI crawler User-Agents, forwards the request to the origin, and reports eligible page responses to Qwairy.

<Warning>
  Use the **Cloudflare Worker** key created by this setup. A Cloudflare Logpush or Generic HTTP key belongs to a different provider contract and is rejected or discarded by this endpoint.
</Warning>

## Before you start

You need:

* access to **Measure > Crawler Analytics > Settings** for the brand;
* a Cloudflare Worker attached to the hostname configured for that brand;
* permission to create an encrypted Worker secret and attach a production route;
* a test page that returns `text/html` or `text/plain`.

Only activate one delivery source for a brand. Running this Worker alongside another active collector can duplicate observations.

## Create and protect the key

1. In Crawler Analytics settings, select **Cloudflare Worker**.
2. Select **Create Key**.
3. Enter a descriptive name and the IANA time zone used for daily analytics.
4. Copy the plaintext secret when it appears. Qwairy shows only its prefix later.
5. Store it as the encrypted Worker secret `QWAIRY_API_KEY`.

Do not put the key in Worker source, ordinary environment variables, request URLs, browser code, or logs. Leave `QWAIRY_ENDPOINT` unset unless Qwairy explicitly gives you a different endpoint.

## Deploy the generated Worker

Create `worker.js` with the recipe generated by the current Qwairy setup:

```javascript theme={null}
/**
 * Qwairy Crawler Analytics - Cloudflare Worker
 *
 * Setup:
 * 1. Create a new Worker in Cloudflare dashboard
 * 2. Set environment variables: QWAIRY_API_KEY, QWAIRY_ENDPOINT (optional)
 * 3. Deploy to a route (e.g., yourdomain.com/*)
 */

const IGNORE_EXTENSIONS = /\.(?:png|jpe?g|gif|svg|webp|ico|mp4|webm|css|js|json|xml|woff2?|ttf|eot|map)$/i;
const IGNORE_PATHS = [/^\/api\//i, /^\/dashboard\//i, /^\/_next\//i, /^\/favicon/i];
const AI_CRAWLER_UA = /(?:^|[^a-z0-9_-])(?:Claude-SearchBot|MistralAI-Index|OAI-SearchBot|PerplexityBot|ChatGPT-User|Google-GeminiNotebook|Google-NotebookLM|Google-Agent|MistralAI-User|Perplexity-User|Claude-User|Google-CloudVertexBot|ClaudeBot|GPTBot|GrokBot|meta-webindexer|Amzn-User|Meta-ExternalFetcher|AI2Bot|Ai2Bot-Dolma|Amazonbot|Bytespider|CCBot|Meta-ExternalAgent|DuckAssistBot|KimiBot|Kimi-SearchBot|YouBot|Kimi-User|Diffbot|Kangaroo Bot|omgili|omgilibot|PanguBot|Timpibot|Webzio-Extended)(?=$|[^a-z0-9_-])/i;

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);
    const pathname = url.pathname;
    const userAgent = request.headers.get('User-Agent') || '';

    if (!AI_CRAWLER_UA.test(userAgent) || IGNORE_EXTENSIONS.test(pathname) || IGNORE_PATHS.some(r => r.test(pathname))) {
      return fetch(request);
    }

    const response = await fetch(request);
    const contentType = response.headers.get('content-type')?.split(';')[0];
    if (!contentType || !['text/html', 'text/plain'].includes(contentType)) {
      return response;
    }

    const apiKey = env.QWAIRY_API_KEY;
    const endpoint = env.QWAIRY_ENDPOINT || 'https://www.qwairy.co/api/v1/logs/cloudflare';
    if (!apiKey) return response;

    ctx.waitUntil(
      fetch(endpoint, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json', 'X-API-Key': apiKey },
        body: JSON.stringify({
          status_code: response.status,
          request_method: request.method,
          request_path: pathname,
          hostname: request.headers.get('Host') || url.hostname,
          user_agent: userAgent,
          timestamp: new Date().toISOString(),
        }),
      }).catch(err => console.error('Qwairy: Failed to send log', err))
    );

    return response;
  },
};
```

Then:

1. Paste the recipe into the Worker.
2. Add `QWAIRY_API_KEY` as an encrypted secret with the one-time Qwairy value.
3. Attach the Worker to the exact production route to monitor, such as `www.example.com/*`.
4. Deploy the Worker and confirm normal requests still receive the origin response.

The generated recipe sends only the path, not the query string. Review your path design and add exclusions in Qwairy if path segments can contain sensitive identifiers.

## Verify delivery

Use a dedicated page that returns HTML and replace the example hostname. This synthetic request creates a crawler observation in the current analytic day:

```bash theme={null}
curl --user-agent "Mozilla/5.0 (compatible; GPTBot/1.1)" \
  --output /dev/null \
  --write-out "%{http_code}\n" \
  "https://www.example.com/docs/qwairy-collector-check"
```

1. Confirm the command returns the expected origin status.
2. In Qwairy, wait for the connector to move from pending to connected after the accepted event reaches a daily rollup.
3. Open the current analytic day and look for `/docs/qwairy-collector-check`.
4. If the connector stays pending, inspect Worker logs for `Qwairy: Failed to send log`, then confirm the route, secret, hostname, User-Agent, and response content type.

An origin response alone proves only that the Worker proxy path works. It does not prove that Qwairy accepted and rolled up the event.

## Limits and behavior

* The recipe reports only matching User-Agents on non-ignored paths whose origin response is `text/html` or `text/plain`.
* The recipe can attempt delivery for other HTTP methods, but Qwairy accepts only `GET` and `HEAD` records for aggregation.
* Static file extensions, `/api/`, `/dashboard/`, `/_next/`, and `/favicon` paths are ignored by the generated code.
* The Worker sends one Qwairy request per matching origin response. It has no delivery queue or retry loop, so network failures can create gaps.
* The integration route allows a burst of 120 ingestion requests per minute. The route can acknowledge over-limit Worker deliveries without aggregating them, so check coverage warnings after traffic spikes.
* Qwairy applies its maintained crawler registry, hostname checks, exclusions, late-arrival window, and ingestion ceilings after receipt.
* The shared technical ceilings are 10,000,000 events per integration per day and 200,000,000 events per billed team per day.
* The collector observes requests. It does not prove indexing, model training, or citation use.

## Rotate or roll back

To rotate the key, create a replacement in Qwairy, update the encrypted Worker secret, deploy, verify a new accepted observation, and then delete the previous key.

To roll back the collector:

1. Detach the Worker route or disable the Worker deployment.
2. Confirm the origin still serves normal traffic and no new Worker observations arrive.
3. Remove the Worker secret.
4. Revoke the corresponding Qwairy key.

Disabling delivery creates an analytics coverage gap. Check delivery-continuity warnings before comparing periods that cross the rollback.

## Related pages

* [Crawler Analytics](/documentation/measure/crawler-analytics)
* [Cloudflare Logpush](/documentation/measure/crawler-analytics/connectors/cloudflare)