> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qwairy.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Netlify Log Drains

> Connect Netlify traffic logs to Crawler Analytics through a General HTTP log drain.

Use a Netlify General HTTP log drain to send site traffic records to Crawler Analytics.

## Before you start

You need:

* a Netlify Enterprise account with permission to configure the target site;
* a site serving the hostname configured for the Qwairy brand;
* access to **Measure > Crawler Analytics > Settings**;
* a secure place to retain the one-time connector secret.

## Create and protect the key

1. In Crawler Analytics settings, select **Netlify**.
2. Select **Create Key**.
3. Enter a name and the IANA time zone used for daily snapshots.
4. Copy the secret when it appears. Qwairy shows only its prefix later.
5. Keep the secret in the Netlify authorization setting. Do not add it to the endpoint query string, source control, or browser code.

For rotation, create a replacement, update the Netlify authorization value, confirm delivery, and then delete the previous key. An integration can have up to five active keys.

## Configure the log drain

1. In the target Netlify site, open **Logs & Metrics > Log Drains**.

2. Select **Enable a log drain** and choose **General HTTP endpoint**.

3. Select **Traffic logs**. Do not add Function, Edge Function, or Deploy logs for this connector.

4. Leave **Exclude personally identifiable information (PII)** unchecked. Crawler classification requires the `user_agent` field.

5. Set the full endpoint URL to:

   ```text theme={null}
   https://www.qwairy.co/api/v1/logs/netlify
   ```

6. Set the authorization header value to:

   ```text theme={null}
   Bearer YOUR_CRAWLER_KEY
   ```

7. Choose **JSON** or **NDJSON** as the Log Drain Format.

8. Select **Connect**.

Netlify's traffic payload can include `client_ip`. Qwairy ignores that field before aggregation and does not retain raw payloads, query parameters, or human requests.

See [Netlify Log Drains](https://docs.netlify.com/manage/monitoring/log-drains/) for the current provider workflow.

## Test delivery

Request a safe path on the monitored site with a recognized crawler User-Agent:

```bash theme={null}
curl --head --user-agent "GPTBot/1.0" "https://www.example.com/docs/crawler-check"
```

Check that Netlify reports a successful drain delivery. In Qwairy, the connector changes from pending only after an accepted traffic row contains a matching hostname, current timestamp, `GET` or `HEAD` method, path, and recognized crawler User-Agent.

## Limits, filtering, and cost

* Netlify's General HTTP drain sends the selected traffic stream and does not expose a crawler-only filter for this integration.
* Human and unknown User-Agent rows are discarded by Qwairy, but they still contribute to Netlify delivery volume and the Qwairy ingress ceilings.
* Review current Netlify log-drain and traffic-delivery pricing before enabling the integration on a high-volume site.
* If the PII exclusion removes `user_agent`, Qwairy cannot classify the request and discards it.
* The shared integration and billed-team daily ceilings are documented on the Crawler Analytics overview.

## Troubleshooting

* **The drain cannot connect**: verify the full HTTPS endpoint and the `Bearer ` prefix before the current key.
* **Qwairy stays pending**: confirm that **Traffic logs** are selected and that the PII exclusion is off, then generate a current request with a recognized crawler User-Agent.
* **Rows are acknowledged but not counted**: check that `log_type` is `traffic`, the URL hostname matches the brand, the method is `GET` or `HEAD`, and the event has a valid timestamp.
* **Counts are lower than source traffic**: human, traditional-search, unknown, excluded, invalid, or out-of-scope rows do not enter crawler aggregates.
* **Delivery volume is unexpectedly high**: Netlify forwards the full selected traffic stream. Reassess provider cost before leaving the connector enabled.

## Related page

* [Crawler Analytics](/documentation/measure/crawler-analytics)