Fastly HTTPS Logging

​

Before you start

• permission to edit and activate the target Fastly service;
• a service serving the hostname configured for the Qwairy brand;
• access to Measure > Crawler Analytics > Settings;
• a secure place to retain the one-time connector secret.

​

Create and protect the key

1. In Crawler Analytics settings, select Fastly.
2. Select Create Key.
3. Enter a name and the IANA time zone used for daily snapshots.
4. Copy the secret when it appears. It cannot be displayed again.
5. Keep it only in Fastly’s custom header value or another restricted secret store.

​

Configure the HTTPS endpoint

1. In the target Fastly service, add an HTTPS logging endpoint.
2. Set the delivery URL to:
   https://www.qwairy.co/api/v1/logs/fastly
   
3. Fastly validates ownership through this public challenge endpoint:
   https://www.qwairy.co/.well-known/fastly/logging/challenge
   
   The challenge authorizes domain validation only. Log delivery still requires the connector key.
4. Use POST, set the content type to application/json, and add this custom header:
   X-API-Key: YOUR_CRAWLER_KEY
   
5. Set Message type to blank.
6. Set Request max entries to 10000 and Request max bytes to 2000000.
7. Choose either an array of JSON or newline-delimited JSON. gzip compression is supported.

​

Use the minimal log format

{
  "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S.000Z"\}, time.start)}V",
  "host": "%{json.escape(req.http.Host)}V",
  "request_path": "%{json.escape(req.url.path)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_status": %{resp.status}V
}

​

Test delivery

curl --head --user-agent "GPTBot/1.0" "https://www.example.com/docs/crawler-check"

​

Limits, filtering, and cost

• Keep the configured request boundaries at 10,000 entries and 2,000,000 bytes.
• Qwairy accepts JSON arrays, NDJSON, and gzip for this route and splits valid deliveries into bounded internal claims.
• Upstream filtering reduces delivered volume. Sampling or a condition that omits crawler requests makes counts partial for the excluded scope.
• Without an upstream condition, Fastly forwards other requests and Qwairy discards them after receipt. Those rows can still affect provider delivery volume and cost.
• Shared daily ingestion ceilings also apply.

​

Troubleshooting

• Endpoint validation fails: confirm that the URL uses www.qwairy.co and that Fastly can reach the displayed well-known challenge path.
• 401: check the exact X-API-Key header name and the current Fastly connector secret.
• 413: restore the 2,000,000-byte request maximum or reduce the delivery size.
• Logs are rejected or ignored: keep Message type blank, use valid JSON or NDJSON, and preserve the timestamp, host, path, method, User-Agent, and status fields from the minimal format.
• Qwairy stays pending: activate the service version and generate a current GET or HEAD request with a recognized crawler User-Agent on the configured hostname.

​

Related page

• Crawler Analytics