Skip to main content
Use CloudFront Standard Logging v2 with Amazon Data Firehose to deliver a minimal JSON request dataset to Crawler Analytics.

​
Before you start

You need:
  • a CloudFront distribution serving the hostname configured for the Qwairy brand;
  • permission to configure Standard Logging v2, Firehose, IAM, CloudWatch, and an S3 failure-backup bucket;
  • access to Measure > Crawler Analytics > Settings;
  • a secure place to retain the one-time connector secret.
CloudFront requires its Firehose log-delivery destination in us-east-1.

​
Create and protect the key

  1. In Crawler Analytics settings, select AWS CloudFront.
  2. Select Create Key.
  3. Enter a name and the IANA time zone used for daily snapshots.
  4. Copy the secret when it appears. Qwairy shows only its prefix later.
  5. Store it as the Firehose HTTP endpoint access key. Do not add it to a URL, source repository, or CloudWatch log message.
To rotate the key, create a replacement, update Firehose, confirm delivery, and then delete the previous key. An integration can have up to five active keys.

​
Configure Firehose

  1. In us-east-1, create an Amazon Data Firehose delivery stream with an HTTP endpoint destination.
  2. Set the endpoint URL to:
  3. Use the Qwairy connector secret as the Firehose Access key. Firehose adds the required X-Amz-Firehose-Access-Key header.
  4. Set the HTTP buffer size to 1 MiB and the interval to 60 seconds. Shorter delivery intervals are not supported by this connector.
  5. Enable S3 backup for failed HTTP deliveries and enable Firehose error logging in CloudWatch.
  6. Grant the Firehose role access to the backup bucket and required log resources.

​
Configure Standard Logging v2

  1. In CloudFront, create or edit a Standard Logging v2 configuration for the distribution.
  2. Select JSON output.
  3. Select the Firehose stream as the delivery destination.
  4. Use exactly this field list:
    This list excludes client address, query, cookie, and referrer fields.
  5. Enable the logging configuration.
See CloudFront Standard Logging v2 and Firehose HTTP backup settings for the current AWS controls.

​
Test delivery

After enabling both configurations, request a safe path on the distribution hostname with a recognized crawler User-Agent:
Check the Firehose HTTP delivery metrics and CloudWatch error log. A successful Qwairy response includes the Firehose request ID. The connector changes from pending after an accepted CloudFront row reaches the rollup pipeline. Generic Firehose demo records do not use the CloudFront Standard Logging v2 shape and therefore do not prove that crawler aggregation is configured correctly.

​
Limits, delivery, and cost

  • Keep the HTTP buffer at 1 MiB and 60 seconds. This keeps the base64 Firehose envelope inside Qwairy’s 2,000,000-byte wire boundary.
  • A delivery can contain at most 10,000 Firehose records or parsed occurrences for this route.
  • Upstream exclusions make the resulting crawler counts partial. Compare only equivalent CloudFront and Qwairy scopes.
  • CloudFront logging, Firehose delivery, CloudWatch error logs, and S3 backup can each incur AWS charges.
  • Shared Qwairy daily ingestion ceilings also apply.

​
Troubleshooting

  • 401 from the HTTP endpoint: confirm that Firehose uses the current Qwairy secret as its Access key.
  • 400 invalid delivery request: verify that the source is CloudFront Standard Logging v2, output is JSON, the exact field list is selected, and the Firehose request ID is preserved.
  • 413 payload too large: restore the 1 MiB Firehose buffer and check that no transformation expands individual records.
  • 415 unsupported encoding: use the Firehose HTTP delivery encoding supported by the setup and keep the destination content type as JSON.
  • Firehose retries or writes to S3 backup: inspect CloudWatch delivery errors, IAM permissions, endpoint settings, and buffering hints.
  • Qwairy stays pending: confirm that logging is enabled on the intended distribution and generate a current GET or HEAD request with a recognized crawler User-Agent to the configured hostname.

​
Related page