Cloudflare Worker

​

Before you start

• access to Measure > Crawler Analytics > Settings for the brand;
• a Cloudflare Worker attached to the hostname configured for that brand;
• permission to create an encrypted Worker secret and attach a production route;
• a test page that returns text/html or text/plain.

​

Create and protect the key

1. In Crawler Analytics settings, select Cloudflare Worker.
2. Select Create Key.
3. Enter a descriptive name and the IANA time zone used for daily analytics.
4. Copy the plaintext secret when it appears. Qwairy shows only its prefix later.
5. Store it as the encrypted Worker secret QWAIRY_API_KEY.

​

Deploy the generated Worker

/**
 * Qwairy Crawler Analytics - Cloudflare Worker
 *
 * Setup:
 * 1. Create a new Worker in Cloudflare dashboard
 * 2. Set environment variables: QWAIRY_API_KEY, QWAIRY_ENDPOINT (optional)
 * 3. Deploy to a route (e.g., yourdomain.com/*)
 */

const IGNORE_EXTENSIONS = /\.(?:png|jpe?g|gif|svg|webp|ico|mp4|webm|css|js|json|xml|woff2?|ttf|eot|map)$/i;
const IGNORE_PATHS = [/^\/api\//i, /^\/dashboard\//i, /^\/_next\//i, /^\/favicon/i];
const AI_CRAWLER_UA = /(?:^|[^a-z0-9_-])(?:Claude-SearchBot|MistralAI-Index|OAI-SearchBot|PerplexityBot|ChatGPT-User|Google-GeminiNotebook|Google-NotebookLM|Google-Agent|MistralAI-User|Perplexity-User|Claude-User|Google-CloudVertexBot|ClaudeBot|GPTBot|GrokBot|meta-webindexer|Amzn-User|Meta-ExternalFetcher|AI2Bot|Ai2Bot-Dolma|Amazonbot|Bytespider|CCBot|Meta-ExternalAgent|DuckAssistBot|KimiBot|Kimi-SearchBot|YouBot|Kimi-User|Diffbot|Kangaroo Bot|omgili|omgilibot|PanguBot|Timpibot|Webzio-Extended)(?=$|[^a-z0-9_-])/i;

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);
    const pathname = url.pathname;
    const userAgent = request.headers.get('User-Agent') || '';

    if (!AI_CRAWLER_UA.test(userAgent) || IGNORE_EXTENSIONS.test(pathname) || IGNORE_PATHS.some(r => r.test(pathname))) {
      return fetch(request);
    }

    const response = await fetch(request);
    const contentType = response.headers.get('content-type')?.split(';')[0];
    if (!contentType || !['text/html', 'text/plain'].includes(contentType)) {
      return response;
    }

    const apiKey = env.QWAIRY_API_KEY;
    const endpoint = env.QWAIRY_ENDPOINT || 'https://www.qwairy.co/api/v1/logs/cloudflare';
    if (!apiKey) return response;

    ctx.waitUntil(
      fetch(endpoint, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json', 'X-API-Key': apiKey },
        body: JSON.stringify({
          status_code: response.status,
          request_method: request.method,
          request_path: pathname,
          hostname: request.headers.get('Host') || url.hostname,
          user_agent: userAgent,
          timestamp: new Date().toISOString(),
        }),
      }).catch(err => console.error('Qwairy: Failed to send log', err))
    );

    return response;
  },
};

1. Paste the recipe into the Worker.
2. Add QWAIRY_API_KEY as an encrypted secret with the one-time Qwairy value.
3. Attach the Worker to the exact production route to monitor, such as www.example.com/*.
4. Deploy the Worker and confirm normal requests still receive the origin response.

​

Verify delivery

curl --user-agent "Mozilla/5.0 (compatible; GPTBot/1.1)" \
  --output /dev/null \
  --write-out "%{http_code}\n" \
  "https://www.example.com/docs/qwairy-collector-check"

1. Confirm the command returns the expected origin status.
2. In Qwairy, wait for the connector to move from pending to connected after the accepted event reaches a daily rollup.
3. Open the current analytic day and look for /docs/qwairy-collector-check.
4. If the connector stays pending, inspect Worker logs for Qwairy: Failed to send log, then confirm the route, secret, hostname, User-Agent, and response content type.

​

Limits and behavior

• The recipe reports only matching User-Agents on non-ignored paths whose origin response is text/html or text/plain.
• The recipe can attempt delivery for other HTTP methods, but Qwairy accepts only GET and HEAD records for aggregation.
• Static file extensions, /api/, /dashboard/, /_next/, and /favicon paths are ignored by the generated code.
• The Worker sends one Qwairy request per matching origin response. It has no delivery queue or retry loop, so network failures can create gaps.
• The integration route allows a burst of 120 ingestion requests per minute. The route can acknowledge over-limit Worker deliveries without aggregating them, so check coverage warnings after traffic spikes.
• Qwairy applies its maintained crawler registry, hostname checks, exclusions, late-arrival window, and ingestion ceilings after receipt.
• The shared technical ceilings are 10,000,000 events per integration per day and 200,000,000 events per billed team per day.
• The collector observes requests. It does not prove indexing, model training, or citation use.

​

Rotate or roll back

1. Detach the Worker route or disable the Worker deployment.
2. Confirm the origin still serves normal traffic and no new Worker observations arrive.
3. Remove the Worker secret.
4. Revoke the corresponding Qwairy key.

​

Related pages

• Crawler Analytics
• Cloudflare Logpush