> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qwairy.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Fastly HTTPS Logging

> Configure a bounded Fastly HTTPS logging endpoint for Crawler Analytics.

Use Fastly HTTPS Logging to deliver a minimal request record from a Fastly CDN service to Crawler Analytics.

## Before you start

You need:

* permission to edit and activate the target Fastly service;
* a service serving the hostname configured for the Qwairy brand;
* access to **Measure > Crawler Analytics > Settings**;
* a secure place to retain the one-time connector secret.

## Create and protect the key

1. In Crawler Analytics settings, select **Fastly**.
2. Select **Create Key**.
3. Enter a name and the IANA time zone used for daily snapshots.
4. Copy the secret when it appears. It cannot be displayed again.
5. Keep it only in Fastly's custom header value or another restricted secret store.

To rotate the key, create a replacement, update the Fastly endpoint, activate the new service version, confirm delivery, and then delete the previous key. An integration can have up to five active keys.

## Configure the HTTPS endpoint

1. In the target Fastly service, add an **HTTPS** logging endpoint.

2. Set the delivery URL to:

   ```text theme={null}
   https://www.qwairy.co/api/v1/logs/fastly
   ```

3. Fastly validates ownership through this public challenge endpoint:

   ```text theme={null}
   https://www.qwairy.co/.well-known/fastly/logging/challenge
   ```

   The challenge authorizes domain validation only. Log delivery still requires the connector key.

4. Use `POST`, set the content type to `application/json`, and add this custom header:

   ```text theme={null}
   X-API-Key: YOUR_CRAWLER_KEY
   ```

5. Set **Message type** to blank.

6. Set **Request max entries** to `10000` and **Request max bytes** to `2000000`.

7. Choose either an array of JSON or newline-delimited JSON. gzip compression is supported.

## Use the minimal log format

Paste this format into the HTTPS logging endpoint:

```text theme={null}
{
  "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S.000Z"\}, time.start)}V",
  "host": "%{json.escape(req.http.Host)}V",
  "request_path": "%{json.escape(req.url.path)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_status": %{resp.status}V
}
```

This format omits the client address, query string, cookies, and referrer. If your Fastly service already has a condition for known AI crawler user agents, apply it before delivery. Qwairy still performs authoritative classification on receipt.

Activate the updated service version on Production. See [Fastly HTTPS log streaming](https://www.fastly.com/documentation/guides/integrations/logging-endpoints/protocol-based-and-self-hosted/log-streaming-https/) for the current provider controls.

## Test delivery

First confirm that Fastly accepts the domain-control challenge. Then request a safe path on the monitored hostname:

```bash theme={null}
curl --head --user-agent "GPTBot/1.0" "https://www.example.com/docs/crawler-check"
```

Check Fastly's endpoint delivery status. Qwairy moves the connector from pending after an accepted log row reaches the rollup pipeline; a successful challenge alone does not connect it.

## Limits, filtering, and cost

* Keep the configured request boundaries at 10,000 entries and 2,000,000 bytes.
* Qwairy accepts JSON arrays, NDJSON, and gzip for this route and splits valid deliveries into bounded internal claims.
* Upstream filtering reduces delivered volume. Sampling or a condition that omits crawler requests makes counts partial for the excluded scope.
* Without an upstream condition, Fastly forwards other requests and Qwairy discards them after receipt. Those rows can still affect provider delivery volume and cost.
* Shared daily ingestion ceilings also apply.

## Troubleshooting

* **Endpoint validation fails**: confirm that the URL uses `www.qwairy.co` and that Fastly can reach the displayed well-known challenge path.
* **401**: check the exact `X-API-Key` header name and the current Fastly connector secret.
* **413**: restore the 2,000,000-byte request maximum or reduce the delivery size.
* **Logs are rejected or ignored**: keep Message type blank, use valid JSON or NDJSON, and preserve the timestamp, host, path, method, User-Agent, and status fields from the minimal format.
* **Qwairy stays pending**: activate the service version and generate a current `GET` or `HEAD` request with a recognized crawler User-Agent on the configured hostname.

## Related page

* [Crawler Analytics](/documentation/measure/crawler-analytics)