> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qwairy.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Apache log files

> Send AI crawler requests from an Apache server to Crawler Analytics: write a dedicated JSON log, send closed files in batches, and verify delivery.

Use this guide when your site runs on an Apache server that you or your host administer, and no CDN or platform connector applies. Apache writes AI crawler requests to a dedicated log file in the format Qwairy expects. A scheduled script then sends the closed files to the [Generic HTTP collector](https://docs.qwairy.co/documentation/measure/crawler-analytics/connectors/generic-http.md).

The dedicated file is added next to your existing access logs, which keep working when you follow the virtual host step. Qwairy receives only the method, path, hostname, User-Agent, time, and status of AI crawler requests.

> [!WARNING]
>
> If a CDN or full-page cache answers some requests before Apache, those requests never reach Apache logs. Collect at that layer instead, or accept partial counts.

## Before you start

If you are not the server administrator, send this list to the person who manages the server. They need:

* Apache 2.4 with `mod_rewrite`, `mod_setenvif`, and `mod_log_config` enabled, and the `rotatelogs` program installed with Apache;
* permission to edit the Apache configuration and reload Apache;
* a Linux shell with `sh`, `curl`, `find`, `split`, and `sha256sum`, and a way to run a command every five minutes, such as cron;
* outbound HTTPS access from the server to `https://www.qwairy.co`;
* the `ServerName` of the virtual host, which must be the hostname configured for the Qwairy brand or its apex or `www` equivalent;
* a key created in Qwairy, as described below.

Only activate one Crawler Analytics delivery source for the brand.

## Create and protect the key

1. In **Measure > Crawler Analytics > Settings**, select **Generic HTTP API**.
2. Select **Create Key**, then enter a descriptive name and the IANA time zone used for daily analytics.
3. Copy the secret when it appears. Qwairy shows only its prefix later.
4. On the server, store it in a file that only the user running the script can read:

```bash theme={null}
sudo install -d -m 700 /etc/qwairy
printf 'export QWAIRY_CRAWLER_KEY=%s\n' 'YOUR_CRAWLER_KEY' | sudo tee /etc/qwairy/crawler.env > /dev/null
sudo chmod 600 /etc/qwairy/crawler.env
```

Replace `YOUR_CRAWLER_KEY` with the secret. Do not commit it to source control or write it to logs.

## Write a dedicated log file

The configuration below writes one JSON event per line, only for `GET` and `HEAD` requests from known AI crawlers. A new file starts every five minutes, so the sender only reads files that Apache has closed.

**1. Create the log directory**

```bash theme={null}
sudo install -d -m 750 /var/log/apache2/qwairy
```

Use the log directory of your distribution, for example `/var/log/httpd` on Red Hat-based systems, and keep the same path in every step.

**2. Define the crawler filter and the log format**

Add these lines to the main server configuration, outside any `<VirtualHost>` block:

```apache theme={null}
SetEnvIfNoCase User-Agent "(?:^|[^a-z0-9_-])(?:Claude-SearchBot|MistralAI-Index|OAI-SearchBot|PerplexityBot|ChatGPT-User|Google-GeminiNotebook|Google-NotebookLM|Google-Agent|MistralAI-User|Perplexity-User|Claude-User|Google-CloudVertexBot|ClaudeBot|GPTBot|GrokBot|meta-webindexer|Amzn-User|Meta-ExternalFetcher|AI2Bot|Ai2Bot-Dolma|Amazonbot|Bytespider|CCBot|Meta-ExternalAgent|DuckAssistBot|KimiBot|Kimi-SearchBot|YouBot|Kimi-User|Diffbot|Kangaroo Bot|omgili|omgilibot|PanguBot|Timpibot|Webzio-Extended)(?=$|[^a-z0-9_-])" qwairy_ai_crawler

LogFormat "{\"request_method\":\"%m\",\"request_path\":\"%<{QWAIRY_PATH}e\",\"hostname\":\"%v\",\"user_agent\":\"%{User-Agent}i\",\"timestamp\":\"%{%Y-%m-%dT%H:%M:%S%z}t\",\"status_code\":%>s}" qwairy_ndjson
```

The User-Agent list matches the recipes generated in Qwairy. Qwairy applies its maintained registry again on receipt, so an older list only misses identities added later.

**3. Log AI crawler requests in the virtual host**

Add these lines inside the `<VirtualHost>` that serves the site over HTTPS, before any other rewrite rule.

If this virtual host has no `CustomLog` line yet, it writes to the main server access log. A `CustomLog` added inside a virtual host stops that inheritance, so first copy the main `CustomLog` line into the virtual host to keep your usual access log.

```apache theme={null}
RewriteEngine On
RewriteCond %{THE_REQUEST} "^[A-Z]+ (/[^?# ]*)"
RewriteRule ^ - [E=QWAIRY_PATH:%1]
CustomLog "|/usr/bin/rotatelogs -l /var/log/apache2/qwairy/qwairy-ai.%Y%m%d%H%M%S.ndjson 300" qwairy_ndjson "expr=-n reqenv('qwairy_ai_crawler') && %{REQUEST_METHOD} in {'GET','HEAD'} && %{THE_REQUEST} =~ /^[ -~]+$/ && %{HTTP:User-Agent} =~ /^[ -~]{1,512}$/"
```

Run `command -v rotatelogs` and use the path it returns. If another virtual host also serves pages of the same site, add the same lines there with a different file prefix.

**4. Check and reload Apache**

```bash theme={null}
sudo apachectl configtest
sudo apachectl graceful
```

Each choice in this configuration avoids a known error:

| Choice | Why |
| - | - |
| `%<{QWAIRY_PATH}e`, read from the request line | Records the path the crawler requested, without query string, before Apache maps `/` to an index file. It also keeps encoded characters such as `%C3%A9`, which keeps each line valid JSON. |
| Printable ASCII conditions on `THE_REQUEST` and `User-Agent`, with a User-Agent of at most 512 characters | Apache writes other bytes as `\x` escapes, which are not valid JSON. Such requests are not logged, so they cannot block a batch. Known AI crawler User-Agents are much shorter than 512 characters. |
| `%v` | Records the `ServerName` of the virtual host instead of the `Host` header sent by the client. |
| `%{%Y-%m-%dT%H:%M:%S%z}t` | Records local time with its offset, for example `2026-10-30T10:15:30+0100`. The endpoint accepts this form. The default Apache date format is rejected. |
| `%>s` | Records the final status returned to the crawler. |
| `rotatelogs` with `300` | Closes the file every five minutes, so files can be sent without tracking read positions. |

A line looks like this:

```json theme={null}
{"request_method":"GET","request_path":"/guides/getting-started","hostname":"www.example.com","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.3; +https://openai.com/gptbot)","timestamp":"2026-10-30T10:15:30+0100","status_code":200}
```

## Send closed files

Save this script as `/usr/local/bin/qwairy-send-logs`, then run `sudo chmod 700 /usr/local/bin/qwairy-send-logs`:

```sh theme={null}
#!/bin/sh
# Send closed Qwairy log files to Crawler Analytics. Run every 5 minutes.
set -eu

LOG_DIR="${QWAIRY_LOG_DIR:-/var/log/apache2/qwairy}"
ENDPOINT="${QWAIRY_ENDPOINT:-https://www.qwairy.co/api/v1/logs/ingest}"
MAX_DELAY="${QWAIRY_MAX_DELAY:-240}"
: "${QWAIRY_CRAWLER_KEY:?Set QWAIRY_CRAWLER_KEY}"

mkdir -p "$LOG_DIR/failed"

# 1. Run one instance at a time. A lock older than an hour was left by a crash.
LOCK="$LOG_DIR/.lock"
find "$LOG_DIR" -maxdepth 1 -type d -name .lock -mmin +60 -exec rmdir {} \;
mkdir "$LOCK" 2>/dev/null || exit 0
trap 'rmdir "$LOCK"' EXIT

# 2. Wait a random delay, so that servers do not all send at the same minute.
if [ "$MAX_DELAY" -gt 0 ]; then
  sleep "$(( $(od -An -N2 -tu2 /dev/urandom | tr -d ' ') % MAX_DELAY ))"
fi

# 3. Group all closed files (unchanged for more than 5 minutes) into batches of
#    at most 1,000 lines and 1,900,000 bytes. The staging directory lets an
#    interrupted run resume with the same batches, so no line is sent twice.
STAGE="$LOG_DIR/.staging"
if [ -d "$STAGE" ] && { [ ! -d "$STAGE/src" ] || [ -z "$(ls -A "$STAGE/src")" ]; }; then
  rm -rf "$STAGE"
fi
if [ ! -d "$STAGE" ]; then
  closed=$(find "$LOG_DIR" -maxdepth 1 -type f -name '*.ndjson' -mmin +5)
  if [ -n "$closed" ]; then
    mkdir "$STAGE"
    date +%Y%m%d%H%M%S > "$STAGE/name"
    mkdir "$STAGE/src"
    printf '%s\n' "$closed" | while read -r file; do mv "$file" "$STAGE/src/"; done
  fi
fi
if [ -d "$STAGE/src" ] && [ ! -e "$STAGE/done" ]; then
  rm -rf "$STAGE/out"
  mkdir "$STAGE/out"
  awk -v prefix="$STAGE/out/qwairy-$(cat "$STAGE/name").ndjson.part." '
    { size = length($0) + 1 }
    n == 0 || lines >= 1000 || bytes + size > 1900000 {
      if (out != "") close(out)
      n++; lines = 0; bytes = 0; out = sprintf("%s%05d", prefix, n)
    }
    { print > out; lines++; bytes += size }
  ' "$STAGE"/src/*
  touch "$STAGE/done"
fi
if [ -e "$STAGE/done" ]; then
  find "$STAGE/out" -type f -exec mv {} "$LOG_DIR/" \;
  rm -rf "$STAGE"
fi

# 4. Send each batch with a stable idempotency key, so a retry is never counted twice.
failed=0
for part in "$LOG_DIR"/*.ndjson.part.*; do
  [ -e "$part" ] || continue
  key=$( { basename "$part"; cat "$part"; } | sha256sum | cut -d ' ' -f 1 )
  status=$(curl --silent --output /dev/null --write-out '%{http_code}' --max-time 30 \
    --header 'Content-Type: application/json' \
    --header "X-API-Key: $QWAIRY_CRAWLER_KEY" \
    --header "Idempotency-Key: $key" \
    --data-binary "@$part" "$ENDPOINT") || status=000
  case "$status" in
    200) rm "$part" ;;
    400 | 409 | 413 | 415) mv "$part" "$LOG_DIR/failed/"; failed=1 ;;
    *) echo "Qwairy returned $status for $part; retrying on the next run" >&2; exit 1 ;;
  esac
done

if [ "$failed" -ne 0 ]; then
  echo "Some batches were rejected and moved to $LOG_DIR/failed" >&2
  exit 1
fi
```

The script runs one instance at a time and first waits a random delay of up to four minutes, so that servers do not all send at the same minute. It then groups all closed files into batches of at most 1,000 lines and 1,900,000 bytes, so a site with little crawler traffic sends a single request per run. Each batch is sent with an `Idempotency-Key` derived from its content, and a staging directory lets an interrupted run resume with the same batches, so no line is counted twice. It handles responses as follows:

| Response | Action |
| - | - |
| `200` | Deletes the batch. |
| `400`, `409`, `413`, or `415` | Moves the batch to `failed/` for inspection, continues, and exits with an error at the end of the run. A `409` means the batch was already delivered with different content: its earlier delivery is counted. |
| `401`, `403`, `429`, `5xx`, or no response | Stops, keeps the batch, and retries on the next run. The script exits with an error, so cron can report it. |

Schedule the script every five minutes, for example in `/etc/cron.d/qwairy`:

```text theme={null}
*/5 * * * * root . /etc/qwairy/crawler.env && /usr/local/bin/qwairy-send-logs
```

If the log directory differs from `/var/log/apache2/qwairy`, add `QWAIRY_LOG_DIR=/your/path` before the script in the cron line.

To run the script by hand without the random delay, set `QWAIRY_MAX_DELAY=0`:

```bash theme={null}
sudo sh -c '. /etc/qwairy/crawler.env && QWAIRY_MAX_DELAY=0 /usr/local/bin/qwairy-send-logs'
```

## Verify delivery

1. Request a page as an AI crawler, replacing the hostname:

   ```bash theme={null}
   curl --user-agent "Mozilla/5.0 (compatible; GPTBot/1.3; +https://openai.com/gptbot)" \
     --output /dev/null --write-out "%{http_code}\n" \
     "https://www.example.com/"
   ```

2. Confirm that a line appears in the newest file of `/var/log/apache2/qwairy` with `"request_path":"/"`, your `ServerName`, and a time with an offset.

3. Wait at least six minutes, then run the script once by hand with the command above. It prints nothing when every batch was accepted, and `failed/` stays empty.

4. In Qwairy, wait for the connector to move from pending to connected, then open the current analytic day and look for the page.

## Limits and behavior

* Events reach Qwairy about 6 to 20 minutes after the request: up to five minutes until the file closes, six minutes before the script treats it as closed, up to five minutes until the next cron run, and up to four minutes of random delay.
* Events are accepted until 72 hours after the end of their analytic day. If sending stops for longer, older events fall outside the [late-arrival window](https://docs.qwairy.co/documentation/measure/crawler-analytics/connectors/generic-http.md#limits-and-delivery-scope) and are rejected. Historical access logs cannot be imported.
* Only `GET` and `HEAD` requests from the listed AI crawlers, with a printable ASCII request line and a printable ASCII User-Agent of at most 512 characters, are written. Human and other traffic never leaves the server.
* One invalid line makes the whole batch return `400`. The other lines of that batch are not counted until you correct it.
* Requests answered by a CDN or cache before Apache are not observed.

## Troubleshooting

| Symptom | Likely cause | Fix |
| - | - | - |
| No file in the log directory | A module is disabled, the directory is missing, the lines are in another virtual host, or the User-Agent is not listed | Check `apachectl -M`, the directory, and the virtual host that answered the test request. |
| `"request_path":"-"` in lines | The rewrite lines are missing from that virtual host, or `RewriteEngine` is off | Add the three rewrite lines to the same `<VirtualHost>` as `CustomLog`. |
| Files in `failed/`, and the script exits with an error | A line is not valid JSON, or the batch was rejected | For a `400`, correct or remove the invalid line, then move the file back to the log directory. For a `409`, an earlier delivery of this batch is already counted: delete the batch, and do not rename it to resend it. |
| Script exits at once and sends nothing | Another run holds the lock in `.lock` | No action. A lock left by an interrupted run is removed after one hour. |
| Script exits with `401` or `403` | Wrong, inactive, or mismatched key | Check `/etc/qwairy/crawler.env` and the key status in Qwairy. |
| Script exits with `429` or `5xx` | Rate limit or temporary unavailability | No action. The next run retries the same batch. |
| Batches accepted but `processed` is `0` | `ServerName` differs from the brand hostname | Set `ServerName` to the brand hostname or its apex or `www` form. |

## Roll back

1. Remove the Qwairy `CustomLog` and rewrite lines from the virtual host, keep your usual `CustomLog`, then run `sudo apachectl graceful`.
2. Wait at least six minutes so the last file counts as closed, then run the script once more by hand with the command above.
3. Check that the log directory only contains an empty `failed/` directory. Handle any remaining batch before going further.
4. Delete `/etc/cron.d/qwairy`, `/usr/local/bin/qwairy-send-logs`, `/etc/qwairy/crawler.env`, and the log directory.
5. Revoke the key in Qwairy.

Stopping delivery creates a coverage gap. Check delivery-continuity warnings before comparing periods that cross it.

Apache directives verified against the [mod\_log\_config](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html), [mod\_rewrite](https://httpd.apache.org/docs/2.4/mod/mod_rewrite.html), [mod\_setenvif](https://httpd.apache.org/docs/2.4/mod/mod_setenvif.html), and [rotatelogs](https://httpd.apache.org/docs/2.4/programs/rotatelogs.html) documentation on 30 October 2026, with Apache 2.4.

## Related pages

* [Generic HTTP collector](https://docs.qwairy.co/documentation/measure/crawler-analytics/connectors/generic-http.md)
* [Crawler Analytics](https://docs.qwairy.co/documentation/measure/crawler-analytics.md)
* [WordPress and PHP](https://docs.qwairy.co/documentation/measure/crawler-analytics/connectors/wordpress.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.