Cloudflare Logpush

​

Before you start

• a Cloudflare Enterprise zone with Logpush access;
• permission to create the Logpush job;
• a Cloudflare API token with Logs Write permission to apply the required upload bounds;
• access to Measure > Crawler Analytics > Settings;
• a secure place to retain the one-time connector secret.

​

Create and protect the key

1. In Crawler Analytics settings, select Cloudflare Logpush.
2. Select Create Key.
3. Enter a name and the IANA time zone used for daily snapshots.
4. Copy the secret when it appears. Qwairy shows only its prefix later.
5. Keep it in the Logpush destination header configuration. Do not place it in an ordinary query parameter, source control, or client-side code.

​

Create the Logpush job

1. In Cloudflare, open Analytics & Logs > Logs > Add Logpush job for the target zone.
2. Select the HTTP Requests dataset and an HTTP destination.
3. Use this destination configuration, replacing the placeholder with the one-time Qwairy secret:
   https://www.qwairy.co/api/v1/logs/cloudflare?header_Authorization=Bearer%20YOUR_CRAWLER_KEY
   
   Cloudflare converts header_Authorization into an HTTP header. The space after Bearer must remain encoded as %20.
4. Select only these fields:
   ClientRequestHost
   ClientRequestMethod
   ClientRequestPath
   ClientRequestUserAgent
   EdgeResponseStatus
   EdgeStartTimestamp
   
5. Keep the timestamp format as RFC 3339.
6. Before enabling the job, use the Cloudflare Logpush API to set:
   {
     "max_upload_bytes": 5000000,
     "max_upload_records": 10000
   }
   
   The default upload size is outside Qwairy’s bounded route contract.
7. Add this job filter. It deliberately over-matches several crawler families; Qwairy’s registry performs the final classification.
   {"where":{"or":[{"key":"ClientRequestUserAgent","operator":"contains","value":"Bot"},{"key":"ClientRequestUserAgent","operator":"contains","value":"bot"},{"key":"ClientRequestUserAgent","operator":"contains","value":"User"},{"key":"ClientRequestUserAgent","operator":"contains","value":"MistralAI"},{"key":"ClientRequestUserAgent","operator":"contains","value":"Notebook"},{"key":"ClientRequestUserAgent","operator":"contains","value":"Google-Agent"},{"key":"ClientRequestUserAgent","operator":"contains","value":"meta-"},{"key":"ClientRequestUserAgent","operator":"contains","value":"Meta-"},{"key":"ClientRequestUserAgent","operator":"contains","value":"spider"},{"key":"ClientRequestUserAgent","operator":"contains","value":"omgili"},{"key":"ClientRequestUserAgent","operator":"contains","value":"Extended"}]}}
   
8. Keep sampling at 100% for the filtered dataset, then enable the job.

​

Test delivery

curl --head --user-agent "GPTBot/1.0" "https://www.example.com/docs/crawler-check"

​

Limits, sampling, and cost

• Each Logpush upload is limited to 5,000,000 uncompressed bytes and 10,000 records.
• The pre-filter reduces delivery volume but intentionally includes some rows that Qwairy later discards.
• A Cloudflare sampling rate below 100% produces partial Qwairy counts.
• Logpush delivery and retained provider logs may carry Cloudflare charges. Review current pricing for the target zone.
• Shared daily ingestion ceilings also apply.

​

Troubleshooting

• Destination validation fails: confirm the %20 encoding, current secret, HTTPS endpoint, and required upload bounds.
• 401: verify that the destination still sends Authorization: Bearer YOUR_CRAWLER_KEY and that the key belongs to Cloudflare Logpush.
• 400 with too many logs: set max_upload_records to 10000 before enabling the job.
• 413: set max_upload_bytes to 5000000 and check compression or decoded payload size.
• Qwairy stays pending: verify the six selected fields, the User-Agent filter, 100% sampling, and a current request to the configured hostname.
• Counts are lower than Cloudflare logs: compare the same filter and sampling scope, then check Qwairy exclusions and delivery-continuity warnings.

​

Related page

• Crawler Analytics