Skip to main content
Use a Netlify General HTTP log drain to send site traffic records to Crawler Analytics.

​
Before you start

You need:
  • a Netlify Enterprise account with permission to configure the target site;
  • a site serving the hostname configured for the Qwairy brand;
  • access to Measure > Crawler Analytics > Settings;
  • a secure place to retain the one-time connector secret.

​
Create and protect the key

  1. In Crawler Analytics settings, select Netlify.
  2. Select Create Key.
  3. Enter a name and the IANA time zone used for daily snapshots.
  4. Copy the secret when it appears. Qwairy shows only its prefix later.
  5. Keep the secret in the Netlify authorization setting. Do not add it to the endpoint query string, source control, or browser code.
For rotation, create a replacement, update the Netlify authorization value, confirm delivery, and then delete the previous key. An integration can have up to five active keys.

​
Configure the log drain

  1. In the target Netlify site, open Logs & Metrics > Log Drains.
  2. Select Enable a log drain and choose General HTTP endpoint.
  3. Select Traffic logs. Do not add Function, Edge Function, or Deploy logs for this connector.
  4. Leave Exclude personally identifiable information (PII) unchecked. Crawler classification requires the user_agent field.
  5. Set the full endpoint URL to:
  6. Set the authorization header value to:
  7. Choose JSON or NDJSON as the Log Drain Format.
  8. Select Connect.
Netlify’s traffic payload can include client_ip. Qwairy ignores that field before aggregation and does not retain raw payloads, query parameters, or human requests. See Netlify Log Drains for the current provider workflow.

​
Test delivery

Request a safe path on the monitored site with a recognized crawler User-Agent:
Check that Netlify reports a successful drain delivery. In Qwairy, the connector changes from pending only after an accepted traffic row contains a matching hostname, current timestamp, GET or HEAD method, path, and recognized crawler User-Agent.

​
Limits, filtering, and cost

  • Netlify’s General HTTP drain sends the selected traffic stream and does not expose a crawler-only filter for this integration.
  • Human and unknown User-Agent rows are discarded by Qwairy, but they still contribute to Netlify delivery volume and the Qwairy ingress ceilings.
  • Review current Netlify log-drain and traffic-delivery pricing before enabling the integration on a high-volume site.
  • If the PII exclusion removes user_agent, Qwairy cannot classify the request and discards it.
  • The shared integration and billed-team daily ceilings are documented on the Crawler Analytics overview.

​
Troubleshooting

  • The drain cannot connect: verify the full HTTPS endpoint and the Bearer prefix before the current key.
  • Qwairy stays pending: confirm that Traffic logs are selected and that the PII exclusion is off, then generate a current request with a recognized crawler User-Agent.
  • Rows are acknowledged but not counted: check that log_type is traffic, the URL hostname matches the brand, the method is GET or HEAD, and the event has a valid timestamp.
  • Counts are lower than source traffic: human, traditional-search, unknown, excluded, invalid, or out-of-scope rows do not enter crawler aggregates.
  • Delivery volume is unexpectedly high: Netlify forwards the full selected traffic stream. Reassess provider cost before leaving the connector enabled.

​
Related page