> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qwairy.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Generic HTTP collector

> Send filtered server or edge request logs to Crawler Analytics through the generic ingestion endpoint.

Use the Generic HTTP API when you control a server or edge runtime and no managed connector matches your stack. Filter requests before delivery and send only the fields needed for crawler aggregation.

## Before you start

You need:

* access to **Measure > Crawler Analytics > Settings** for the brand;
* a server-side or edge runtime that can observe requests after routing;
* the hostname configured for the Qwairy brand;
* a secure secret store for the connector key.

The collector accepts only `GET` and `HEAD` request events for the configured brand hostname. An apex domain and its `www` form are treated as equivalent. Other subdomains need their own brand scope.

## Create and protect the key

1. In Crawler Analytics settings, select **Generic HTTP API**.
2. Select **Create Key**.
3. Enter a descriptive name and an IANA analytics time zone, such as `Europe/Paris`.
4. Copy the secret when it appears. Qwairy shows only its prefix later.
5. Store it as a server-side secret. Do not place it in browser code, source control, request URLs, or application logs.

To rotate the key, create a replacement, update the collector, confirm delivery, and then delete the previous key. An integration can have up to five active keys.

## Send events

Send `POST` requests to:

```text theme={null}
https://www.qwairy.co/api/v1/logs/ingest
```

Use these headers:

```text theme={null}
Content-Type: application/json
X-API-Key: YOUR_CRAWLER_KEY
```

`Authorization: Bearer YOUR_CRAWLER_KEY` is also accepted. Prefer `X-API-Key` for the setup shown in Qwairy.

Each event used for aggregation needs these fields:

| Field            | Requirement                                                          |
| ---------------- | -------------------------------------------------------------------- |
| `request_method` | `GET` or `HEAD`                                                      |
| `request_path`   | Path for the requested page; query strings and fragments are removed |
| `hostname`       | Configured brand hostname, or its apex/`www` equivalent              |
| `user_agent`     | Full User-Agent used for crawler classification                      |
| `timestamp`      | Valid ISO 8601 event time                                            |
| `status_code`    | Optional HTTP response status                                        |

The API accepts one object, an array of objects, newline-delimited JSON, or an object with a `logs` array. A stable `Idempotency-Key` header or `batch_id` field lets byte-different retries be deduplicated.

Filter known AI crawler user agents in your runtime before calling the endpoint. Qwairy applies its maintained registry again on receipt and discards human, traditional-search, and unknown user agents.

## Test delivery

Replace the hostname and key before running this synthetic check:

```bash theme={null}
event_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
curl --request POST "https://www.qwairy.co/api/v1/logs/ingest" \
  --header "Content-Type: application/json" \
  --header "X-API-Key: $QWAIRY_CRAWLER_KEY" \
  --data "{\"request_method\":\"GET\",\"request_path\":\"/docs/crawler-check\",\"hostname\":\"www.example.com\",\"user_agent\":\"GPTBot/1.0\",\"timestamp\":\"${event_time}\",\"status_code\":200}"
```

A valid request returns a JSON acknowledgement. The connector changes from pending only after the rollup pipeline accepts an event in the configured brand scope. An endpoint health response or a filtered event does not establish that connection.

## Limits and delivery scope

* A generic request can contain at most 1,000 events and 2,000,000 normalized bytes.
* Events delivered more than 72 hours late fall outside the current late-arrival window.
* The shared ceilings are 10,000,000 events per integration per day and 200,000,000 events per billed team per day.
* Your runtime, network provider, or observability stack may charge for reading and forwarding logs.
* Filtering or sampling before delivery makes the resulting counts partial for that scope.

## Troubleshooting

* **401 or 403**: confirm that the secret belongs to the Generic HTTP connector for this brand and is still active.
* **400**: validate the JSON shape, record count, method, path, hostname, User-Agent, and timestamp.
* **413**: reduce the batch size or encoded payload size.
* **429**: pause until the `Retry-After` interval has elapsed, then retry the same idempotent batch.
* **Acknowledged but not connected**: confirm that the hostname matches the brand and that the test uses a currently recognized AI crawler User-Agent and a current timestamp.
* **Counts are lower than source logs**: check upstream filters, sampling, excluded paths, excluded user agents, late delivery, and rejected hostnames.

## Related page

* [Crawler Analytics](/documentation/measure/crawler-analytics)