Skip to main content
Use the Generic HTTP API when you control a server or edge runtime and no managed connector matches your stack. Filter requests before delivery and send only the fields needed for crawler aggregation.

​
Before you start

You need:
  • access to Measure > Crawler Analytics > Settings for the brand;
  • a server-side or edge runtime that can observe requests after routing;
  • the hostname configured for the Qwairy brand;
  • a secure secret store for the connector key.
The collector accepts only GET and HEAD request events for the configured brand hostname. An apex domain and its www form are treated as equivalent. Other subdomains need their own brand scope.

​
Create and protect the key

  1. In Crawler Analytics settings, select Generic HTTP API.
  2. Select Create Key.
  3. Enter a descriptive name and an IANA analytics time zone, such as Europe/Paris.
  4. Copy the secret when it appears. Qwairy shows only its prefix later.
  5. Store it as a server-side secret. Do not place it in browser code, source control, request URLs, or application logs.
To rotate the key, create a replacement, update the collector, confirm delivery, and then delete the previous key. An integration can have up to five active keys.

​
Send events

Send POST requests to:
Use these headers:
Authorization: Bearer YOUR_CRAWLER_KEY is also accepted. Prefer X-API-Key for the setup shown in Qwairy. Each event used for aggregation needs these fields: The API accepts one object, an array of objects, newline-delimited JSON, or an object with a logs array. A stable Idempotency-Key header or batch_id field lets byte-different retries be deduplicated. Filter known AI crawler user agents in your runtime before calling the endpoint. Qwairy applies its maintained registry again on receipt and discards human, traditional-search, and unknown user agents.

​
Test delivery

Replace the hostname and key before running this synthetic check:
A valid request returns a JSON acknowledgement. The connector changes from pending only after the rollup pipeline accepts an event in the configured brand scope. An endpoint health response or a filtered event does not establish that connection.

​
Limits and delivery scope

  • A generic request can contain at most 1,000 events and 2,000,000 normalized bytes.
  • Events delivered more than 72 hours late fall outside the current late-arrival window.
  • The shared ceilings are 10,000,000 events per integration per day and 200,000,000 events per billed team per day.
  • Your runtime, network provider, or observability stack may charge for reading and forwarding logs.
  • Filtering or sampling before delivery makes the resulting counts partial for that scope.

​
Troubleshooting

  • 401 or 403: confirm that the secret belongs to the Generic HTTP connector for this brand and is still active.
  • 400: validate the JSON shape, record count, method, path, hostname, User-Agent, and timestamp.
  • 413: reduce the batch size or encoded payload size.
  • 429: pause until the Retry-After interval has elapsed, then retry the same idempotent batch.
  • Acknowledged but not connected: confirm that the hostname matches the brand and that the test uses a currently recognized AI crawler User-Agent and a current timestamp.
  • Counts are lower than source logs: check upstream filters, sampling, excluded paths, excluded user agents, late delivery, and rejected hostnames.

​
Related page