Fastly

Fastly's real-time log streaming can send your service's request logs to Ahrefs over HTTPS. You add a logging endpoint to your Fastly service from the Fastly control panel; Ahrefs never needs access to your Fastly account.

Prerequisites

• A Fastly CDN service serving the domain of your Ahrefs project. Any Fastly plan works, including the free tier.
• Permission to edit that service's configuration and activate a new version.

Step 1 — Get your access key

1. In Ahrefs, go to Project Settings → Bot Analytics for your project.
2. Select Fastly.
3. Copy the custom header value. It is your access key: it identifies your project, so treat it like a password.

Everything else you need is on this page.

Step 2 — Add the logging endpoint

In the Fastly control panel, open your service, click Edit configuration and clone the active version. In the left menu choose Logging, then HTTPS and Create endpoint.

Endpoint fields

Field	Value
Name	Ahrefs Bot Analytics
Placement	Format Version Default
Log format	The log format below, replacing the default one
URL	https://analytics.ahrefs.com/api/fastly/logs
Maximum logs, Maximum bytes	Leave empty
Content type	application/json
Custom header name	X-Ahrefs-Web-Analytics-Property
Custom header value	The access key from Step 1
Method	POST
JSON log entry format	Newline delimited. Array of JSON also works.
Compression	Gzip (recommended) or none. Zstd and Snappy are not accepted.
TLS fields	Leave empty

Under Advanced options, set Select a log line format to Blank. Click Create, then Activate the new version. Fastly starts streaming right away.

Log format

Fastly fills in the %{...}V placeholders for every request. You decide what to share: the minimal format is enough for Bot Analytics to work, the recommended format gives better results. Unknown keys are ignored, so you can keep extra fields you already log.

Minimal. Bot visits with time, page and bot name:

{
  "timestamp": %{time.start.msec}V,
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "url": "%{json.escape(req.url)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V"
}

Recommended. Adds the visitor IP for location data, the status and response details shown with each visit, Web Bot Auth verification, and the edge marker that removes duplicates when your service uses shielding:

{
  "timestamp": %{time.start.msec}V,
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "scheme": "%{req.protocol}V",
  "url": "%{json.escape(req.url)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_referer": "%{json.escape(req.http.Referer)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_state": "%{json.escape(fastly_info.state)}V",
  "response_status": %{resp.status}V,
  "response_body_size": %{resp.body_bytes_written}V,
  "response_content_type": "%{json.escape(resp.http.Content-Type)}V",
  "fastly_server": "%{json.escape(server.identity)}V",
  "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V,
  "request_id": "%{json.escape(req.xid)}V",
  "signature": "%{json.escape(req.http.Signature)}V",
  "signature_input": "%{json.escape(req.http.Signature-Input)}V",
  "signature_agent": "%{json.escape(req.http.Signature-Agent)}V"
}

Mix the two as you like: the four minimal keys are required, every other key can be left out, empty or (null). Keep fastly_is_edge whenever your service has a shield, otherwise each miss is counted twice. The unquoted keys (timestamp, response_status, response_body_size, fastly_is_edge) must always have a value, or the line is not valid JSON.

Step 3 — Verify

Fastly delivers logs in small batches, usually within a minute of the requests. Bot visits then appear in the Bot Analytics report a few minutes later. Requests from browsers are sent too; Ahrefs discards everything that is not a bot.

Troubleshooting

Fastly does not show delivery errors for HTTPS endpoints in the control panel, so check the settings against the table above when nothing arrives. Ahrefs answers each batch with one of these:

Response	Cause
401 missing authentication header	The custom header name is misspelled or empty.
403 invalid authentication	The custom header value does not match an Ahrefs project. Copy it again from Step 1.
400 record 0: …	The log line format is Classic instead of Blank, the compression is not Gzip or none, or the log format is not valid JSON.
400 record N: Missing record field …	Newline delimited: the log format lacks one of the required keys.
400 invalid body: not a JSON array of log records: …	Array of JSON: the log line format is Classic instead of Blank, the log format lacks a required key, or it is not valid JSON.
400 body is not valid gzip	Compression is Gzip in Fastly but the body could not be decompressed.