Skip to main content
Use Akamai DataStream 2 to send a minimal JSON request dataset to Crawler Analytics through a custom HTTPS destination.

​
Before you start

You need:
  • Akamai DataStream 2 for the target property;
  • permission to change Property Manager logging and activate a stream;
  • access to Measure > Crawler Analytics > Settings;
  • a secure place to retain the one-time connector secret.

​
Create and protect the key

  1. In Crawler Analytics settings, select Akamai.
  2. Select Create Key.
  3. Enter a name and the IANA time zone used for daily snapshots.
  4. Copy the secret when it appears. It cannot be displayed again.
  5. Store it as the Basic authentication password for the custom HTTPS destination.
To rotate the key, create a replacement, update and activate the destination, confirm delivery, and then delete the previous key. An integration can have up to five active keys.

​
Configure the stream

  1. In Akamai Control Center, create or edit a DataStream 2 stream for the target property.
  2. Include exactly these request fields:
  3. Select JSON as the log format.
  4. In Property Manager, enable Log Request Details and Log User-Agent Header so UA is populated.
  5. Set the push frequency to 60 seconds or longer.
  6. Choose a Custom HTTPS destination and set:
  7. If the property can match known AI crawler user agents before streaming, apply that condition. Otherwise keep the dataset minimal and let Qwairy discard unrecognized rows.
  8. Review and activate the stream.
See Akamai’s custom HTTPS destination guide for the current provider workflow.

​
Test delivery

Use Akamai’s destination validation or a mock stream to verify the HTTPS and Basic authentication settings. A validation payload does not necessarily contain an accepted crawler record. After activation, request a safe path on the monitored hostname with a recognized crawler User-Agent:
Check the DataStream delivery status. Qwairy changes the connector from pending after a valid crawler row reaches the rollup pipeline.

​
Limits, sampling, and cost

  • Qwairy accepts Akamai wire deliveries up to 4,000,000 bytes. DataStream can split larger provider files into additional deliveries.
  • Keep the push frequency at 60 seconds or longer for this connector.
  • A DataStream sample below 100%, or a Property Manager rule that excludes requests, makes Qwairy totals partial for that scope.
  • The minimal field list excludes client address, query, cookie, and referrer data.
  • DataStream delivery and destination traffic may carry Akamai charges. Shared Qwairy daily ceilings also apply.

​
Troubleshooting

  • Destination validation fails: confirm HTTPS, application/json, Basic authentication, username qwairy, and the current connector secret.
  • Qwairy stays pending: confirm the stream is active, the push frequency is supported, and UA, reqHost, reqMethod, reqPath, and reqTimeSec are populated.
  • Rows are acknowledged but not counted: only current GET and HEAD rows for the configured hostname with a recognized crawler User-Agent are aggregated.
  • Counts are lower than Akamai reports: check the DataStream sampling rate, Property Manager match criteria, upload errors, Qwairy exclusion rules, and equivalent time windows.
  • Delivery is intermittent: review the destination’s DataStream upload metrics and alerts before interpreting missing periods as zero.

​
Related page