Apache log files

Use this guide when your site runs on an Apache server that you or your host administer, and no CDN or platform connector applies. Apache writes AI crawler requests to a dedicated log file in the format Qwairy expects. A scheduled script then sends the closed files to the Generic HTTP collector. The dedicated file is added next to your existing access logs, which keep working when you follow the virtual host step. Qwairy receives only the method, path, hostname, User-Agent, time, and status of AI crawler requests.

If a CDN or full-page cache answers some requests before Apache, those requests never reach Apache logs. Collect at that layer instead, or accept partial counts.

Before you start

If you are not the server administrator, send this list to the person who manages the server. They need:

Apache 2.4 with mod_rewrite, mod_setenvif, and mod_log_config enabled, and the rotatelogs program installed with Apache;
permission to edit the Apache configuration and reload Apache;
a Linux shell with sh, curl, find, split, and sha256sum, and a way to run a command every five minutes, such as cron;
outbound HTTPS access from the server to https://www.qwairy.co;
the ServerName of the virtual host, which must be the hostname configured for the Qwairy brand or its apex or www equivalent;
a key created in Qwairy, as described below.

Only activate one Crawler Analytics delivery source for the brand.

Create and protect the key

In Measure > Crawler Analytics > Settings, select Generic HTTP API.
Select Create Key, then enter a descriptive name and the IANA time zone used for daily analytics.
Copy the secret when it appears. Qwairy shows only its prefix later.
On the server, store it in a file that only the user running the script can read:

sudo install -d -m 700 /etc/qwairy
printf 'export QWAIRY_CRAWLER_KEY=%s\n' 'YOUR_CRAWLER_KEY' | sudo tee /etc/qwairy/crawler.env > /dev/null
sudo chmod 600 /etc/qwairy/crawler.env

Replace YOUR_CRAWLER_KEY with the secret. Do not commit it to source control or write it to logs.

Write a dedicated log file

The configuration below writes one JSON event per line, only for GET and HEAD requests from known AI crawlers. A new file starts every five minutes, so the sender only reads files that Apache has closed.

Create the log directory

sudo install -d -m 750 /var/log/apache2/qwairy

Use the log directory of your distribution, for example /var/log/httpd on Red Hat-based systems, and keep the same path in every step.

Define the crawler filter and the log format

Add these lines to the main server configuration, outside any <VirtualHost> block:

SetEnvIfNoCase User-Agent "(?:^|[^a-z0-9_-])(?:Claude-SearchBot|MistralAI-Index|OAI-SearchBot|PerplexityBot|ChatGPT-User|Google-GeminiNotebook|Google-NotebookLM|Google-Agent|MistralAI-User|Perplexity-User|Claude-User|Google-CloudVertexBot|ClaudeBot|GPTBot|GrokBot|meta-webindexer|Amzn-User|Meta-ExternalFetcher|AI2Bot|Ai2Bot-Dolma|Amazonbot|Bytespider|CCBot|Meta-ExternalAgent|DuckAssistBot|KimiBot|Kimi-SearchBot|YouBot|Kimi-User|Diffbot|Kangaroo Bot|omgili|omgilibot|PanguBot|Timpibot|Webzio-Extended)(?=$|[^a-z0-9_-])" qwairy_ai_crawler

LogFormat "{\"request_method\":\"%m\",\"request_path\":\"%<{QWAIRY_PATH}e\",\"hostname\":\"%v\",\"user_agent\":\"%{User-Agent}i\",\"timestamp\":\"%{%Y-%m-%dT%H:%M:%S%z}t\",\"status_code\":%>s}" qwairy_ndjson

The User-Agent list matches the recipes generated in Qwairy. Qwairy applies its maintained registry again on receipt, so an older list only misses identities added later.

Log AI crawler requests in the virtual host

Add these lines inside the <VirtualHost> that serves the site over HTTPS, before any other rewrite rule.If this virtual host has no CustomLog line yet, it writes to the main server access log. A CustomLog added inside a virtual host stops that inheritance, so first copy the main CustomLog line into the virtual host to keep your usual access log.

RewriteEngine On
RewriteCond %{THE_REQUEST} "^[A-Z]+ (/[^?# ]*)"
RewriteRule ^ - [E=QWAIRY_PATH:%1]
CustomLog "|/usr/bin/rotatelogs -l /var/log/apache2/qwairy/qwairy-ai.%Y%m%d%H%M%S.ndjson 300" qwairy_ndjson "expr=-n reqenv('qwairy_ai_crawler') && %{REQUEST_METHOD} in {'GET','HEAD'} && %{THE_REQUEST} =~ /^[ -~]+$/ && %{HTTP:User-Agent} =~ /^[ -~]{1,512}$/"

Run command -v rotatelogs and use the path it returns. If another virtual host also serves pages of the same site, add the same lines there with a different file prefix.

Check and reload Apache

sudo apachectl configtest
sudo apachectl graceful

Each choice in this configuration avoids a known error:

Choice	Why
`%<{QWAIRY_PATH}e`, read from the request line	Records the path the crawler requested, without query string, before Apache maps `/` to an index file. It also keeps encoded characters such as `%C3%A9`, which keeps each line valid JSON.
Printable ASCII conditions on `THE_REQUEST` and `User-Agent`, with a User-Agent of at most 512 characters	Apache writes other bytes as `\x` escapes, which are not valid JSON. Such requests are not logged, so they cannot block a batch. Known AI crawler User-Agents are much shorter than 512 characters.
`%v`	Records the `ServerName` of the virtual host instead of the `Host` header sent by the client.
`%{%Y-%m-%dT%H:%M:%S%z}t`	Records local time with its offset, for example `2026-10-30T10:15:30+0100`. The endpoint accepts this form. The default Apache date format is rejected.
`%>s`	Records the final status returned to the crawler.
`rotatelogs` with `300`	Closes the file every five minutes, so files can be sent without tracking read positions.

A line looks like this:

{"request_method":"GET","request_path":"/guides/getting-started","hostname":"www.example.com","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.3; +https://openai.com/gptbot)","timestamp":"2026-10-30T10:15:30+0100","status_code":200}

Send closed files

Save this script as /usr/local/bin/qwairy-send-logs, then run sudo chmod 700 /usr/local/bin/qwairy-send-logs:

#!/bin/sh
# Send closed Qwairy log files to Crawler Analytics. Run every 5 minutes.
set -eu

LOG_DIR="${QWAIRY_LOG_DIR:-/var/log/apache2/qwairy}"
ENDPOINT="${QWAIRY_ENDPOINT:-https://www.qwairy.co/api/v1/logs/ingest}"
MAX_DELAY="${QWAIRY_MAX_DELAY:-240}"
: "${QWAIRY_CRAWLER_KEY:?Set QWAIRY_CRAWLER_KEY}"

mkdir -p "$LOG_DIR/failed"

# 1. Run one instance at a time. A lock older than an hour was left by a crash.
LOCK="$LOG_DIR/.lock"
find "$LOG_DIR" -maxdepth 1 -type d -name .lock -mmin +60 -exec rmdir {} \;
mkdir "$LOCK" 2>/dev/null || exit 0
trap 'rmdir "$LOCK"' EXIT

# 2. Wait a random delay, so that servers do not all send at the same minute.
if [ "$MAX_DELAY" -gt 0 ]; then
  sleep "$(( $(od -An -N2 -tu2 /dev/urandom | tr -d ' ') % MAX_DELAY ))"
fi

# 3. Group all closed files (unchanged for more than 5 minutes) into batches of
#    at most 1,000 lines and 1,900,000 bytes. The staging directory lets an
#    interrupted run resume with the same batches, so no line is sent twice.
STAGE="$LOG_DIR/.staging"
if [ -d "$STAGE" ] && { [ ! -d "$STAGE/src" ] || [ -z "$(ls -A "$STAGE/src")" ]; }; then
  rm -rf "$STAGE"
fi
if [ ! -d "$STAGE" ]; then
  closed=$(find "$LOG_DIR" -maxdepth 1 -type f -name '*.ndjson' -mmin +5)
  if [ -n "$closed" ]; then
    mkdir "$STAGE"
    date +%Y%m%d%H%M%S > "$STAGE/name"
    mkdir "$STAGE/src"
    printf '%s\n' "$closed" | while read -r file; do mv "$file" "$STAGE/src/"; done
  fi
fi
if [ -d "$STAGE/src" ] && [ ! -e "$STAGE/done" ]; then
  rm -rf "$STAGE/out"
  mkdir "$STAGE/out"
  awk -v prefix="$STAGE/out/qwairy-$(cat "$STAGE/name").ndjson.part." '
    { size = length($0) + 1 }
    n == 0 || lines >= 1000 || bytes + size > 1900000 {
      if (out != "") close(out)
      n++; lines = 0; bytes = 0; out = sprintf("%s%05d", prefix, n)
    }
    { print > out; lines++; bytes += size }
  ' "$STAGE"/src/*
  touch "$STAGE/done"
fi
if [ -e "$STAGE/done" ]; then
  find "$STAGE/out" -type f -exec mv {} "$LOG_DIR/" \;
  rm -rf "$STAGE"
fi

# 4. Send each batch with a stable idempotency key, so a retry is never counted twice.
failed=0
for part in "$LOG_DIR"/*.ndjson.part.*; do
  [ -e "$part" ] || continue
  key=$( { basename "$part"; cat "$part"; } | sha256sum | cut -d ' ' -f 1 )
  status=$(curl --silent --output /dev/null --write-out '%{http_code}' --max-time 30 \
    --header 'Content-Type: application/json' \
    --header "X-API-Key: $QWAIRY_CRAWLER_KEY" \
    --header "Idempotency-Key: $key" \
    --data-binary "@$part" "$ENDPOINT") || status=000
  case "$status" in
    200) rm "$part" ;;
    400 | 409 | 413 | 415) mv "$part" "$LOG_DIR/failed/"; failed=1 ;;
    *) echo "Qwairy returned $status for $part; retrying on the next run" >&2; exit 1 ;;
  esac
done

if [ "$failed" -ne 0 ]; then
  echo "Some batches were rejected and moved to $LOG_DIR/failed" >&2
  exit 1
fi

The script runs one instance at a time and first waits a random delay of up to four minutes, so that servers do not all send at the same minute. It then groups all closed files into batches of at most 1,000 lines and 1,900,000 bytes, so a site with little crawler traffic sends a single request per run. Each batch is sent with an Idempotency-Key derived from its content, and a staging directory lets an interrupted run resume with the same batches, so no line is counted twice. It handles responses as follows:

Response	Action
`200`	Deletes the batch.
`400`, `409`, `413`, or `415`	Moves the batch to `failed/` for inspection, continues, and exits with an error at the end of the run. A `409` means the batch was already delivered with different content: its earlier delivery is counted.
`401`, `403`, `429`, `5xx`, or no response	Stops, keeps the batch, and retries on the next run. The script exits with an error, so cron can report it.

Schedule the script every five minutes, for example in /etc/cron.d/qwairy:

*/5 * * * * root . /etc/qwairy/crawler.env && /usr/local/bin/qwairy-send-logs

If the log directory differs from /var/log/apache2/qwairy, add QWAIRY_LOG_DIR=/your/path before the script in the cron line. To run the script by hand without the random delay, set QWAIRY_MAX_DELAY=0:

sudo sh -c '. /etc/qwairy/crawler.env && QWAIRY_MAX_DELAY=0 /usr/local/bin/qwairy-send-logs'

Verify delivery

Request a page as an AI crawler, replacing the hostname:

curl --user-agent "Mozilla/5.0 (compatible; GPTBot/1.3; +https://openai.com/gptbot)" \
  --output /dev/null --write-out "%{http_code}\n" \
  "https://www.example.com/"

Confirm that a line appears in the newest file of /var/log/apache2/qwairy with "request_path":"/", your ServerName, and a time with an offset.
Wait at least six minutes, then run the script once by hand with the command above. It prints nothing when every batch was accepted, and failed/ stays empty.
In Qwairy, wait for the connector to move from pending to connected, then open the current analytic day and look for the page.

Limits and behavior

Events reach Qwairy about 6 to 20 minutes after the request: up to five minutes until the file closes, six minutes before the script treats it as closed, up to five minutes until the next cron run, and up to four minutes of random delay.
Events are accepted until 72 hours after the end of their analytic day. If sending stops for longer, older events fall outside the late-arrival window and are rejected. Historical access logs cannot be imported.
Only GET and HEAD requests from the listed AI crawlers, with a printable ASCII request line and a printable ASCII User-Agent of at most 512 characters, are written. Human and other traffic never leaves the server.
One invalid line makes the whole batch return 400. The other lines of that batch are not counted until you correct it.
Requests answered by a CDN or cache before Apache are not observed.

Troubleshooting

Symptom	Likely cause	Fix
No file in the log directory	A module is disabled, the directory is missing, the lines are in another virtual host, or the User-Agent is not listed	Check `apachectl -M`, the directory, and the virtual host that answered the test request.
`"request_path":"-"` in lines	The rewrite lines are missing from that virtual host, or `RewriteEngine` is off	Add the three rewrite lines to the same `<VirtualHost>` as `CustomLog`.
Files in `failed/`, and the script exits with an error	A line is not valid JSON, or the batch was rejected	For a `400`, correct or remove the invalid line, then move the file back to the log directory. For a `409`, an earlier delivery of this batch is already counted: delete the batch, and do not rename it to resend it.
Script exits at once and sends nothing	Another run holds the lock in `.lock`	No action. A lock left by an interrupted run is removed after one hour.
Script exits with `401` or `403`	Wrong, inactive, or mismatched key	Check `/etc/qwairy/crawler.env` and the key status in Qwairy.
Script exits with `429` or `5xx`	Rate limit or temporary unavailability	No action. The next run retries the same batch.
Batches accepted but `processed` is `0`	`ServerName` differs from the brand hostname	Set `ServerName` to the brand hostname or its apex or `www` form.

Roll back

Remove the Qwairy CustomLog and rewrite lines from the virtual host, keep your usual CustomLog, then run sudo apachectl graceful.
Wait at least six minutes so the last file counts as closed, then run the script once more by hand with the command above.
Check that the log directory only contains an empty failed/ directory. Handle any remaining batch before going further.
Delete /etc/cron.d/qwairy, /usr/local/bin/qwairy-send-logs, /etc/qwairy/crawler.env, and the log directory.
Revoke the key in Qwairy.

Stopping delivery creates a coverage gap. Check delivery-continuity warnings before comparing periods that cross it. Apache directives verified against the mod_log_config, mod_rewrite, mod_setenvif, and rotatelogs documentation on 30 October 2026, with Apache 2.4.

Get started

Monitor

Analyze

Act and optimize

Measure

AI Assistant

Manage workspace

Availability

Team and security

Before you start

Create and protect the key

Write a dedicated log file

Send closed files

Verify delivery

Limits and behavior

Troubleshooting

Roll back

​Before you start

​Create and protect the key

​Write a dedicated log file

​Send closed files

​Verify delivery

​Limits and behavior

​Troubleshooting

​Roll back

​Related pages

Before you start

Create and protect the key

Write a dedicated log file

Send closed files

Verify delivery

Limits and behavior

Troubleshooting

Roll back

Related pages