# Fastly

Fastly's real-time log streaming can send your service's request logs to Ahrefs over HTTPS. You add a logging endpoint to your Fastly service from the Fastly control panel; Ahrefs never needs access to your Fastly account.

## Prerequisites

- A Fastly CDN service serving the domain of your Ahrefs project. Any Fastly plan works, including the free tier.
- Permission to edit that service's configuration and activate a new version.

## Step 1 — Get your access key

1. In Ahrefs, go to **Project Settings → Bot Analytics** for your project.
2. Select **Fastly**.
3. Copy the **custom header value**. It is your access key: it identifies your project, so treat it like a password.

Everything else you need is on this page.

## Step 2 — Add the logging endpoint

In the Fastly control panel, open your service, click **Edit configuration** and clone the active version. In the left menu choose **Logging**, then **HTTPS** and **Create endpoint**.

### Endpoint fields

| Field | Value |
|---|---|
| **Name** | `Ahrefs Bot Analytics` |
| **Placement** | **Format Version Default** |
| **Log format** | The [log format](#log-format) below, replacing the default one |
| **URL** | `https://analytics.ahrefs.com/api/fastly/logs` |
| **Maximum logs**, **Maximum bytes** | Leave empty |
| **Content type** | `application/json` |
| **Custom header name** | `X-Ahrefs-Web-Analytics-Property` |
| **Custom header value** | The access key from Step 1 |
| **Method** | **POST** |
| **JSON log entry format** | **Newline delimited**. **Array of JSON** also works. |
| **Compression** | **Gzip** (recommended) or none. Zstd and Snappy are not accepted. |
| **TLS** fields | Leave empty |

Under **Advanced options**, set **Select a log line format** to **Blank**. Click **Create**, then **Activate** the new version. Fastly starts streaming right away.

> **WARNING**
> The log line format must be **Blank**. The default, **Classic**, prepends a syslog header to every line, and Ahrefs rejects those batches.

> **INFO**
> When you save the endpoint, Fastly checks that the destination host accepts logs from Fastly services. Ahrefs answers that check for everyone, so there is nothing to configure on your side.

### Log format

Fastly fills in the `%{...}V` placeholders for every request. You decide what to share: the minimal format is enough for Bot Analytics to work, the recommended format gives better results. Unknown keys are ignored, so you can keep extra fields you already log.

**Minimal.** Bot visits with time, page and bot name:

```
{
  "timestamp": %{time.start.msec}V,
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "url": "%{json.escape(req.url)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V"
}
```

**Recommended.** Adds the visitor IP for location data, the status and response details shown with each visit, Web Bot Auth verification, and the edge marker that removes duplicates when your service uses shielding:

```
{
  "timestamp": %{time.start.msec}V,
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "scheme": "%{req.protocol}V",
  "url": "%{json.escape(req.url)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_referer": "%{json.escape(req.http.Referer)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_state": "%{json.escape(fastly_info.state)}V",
  "response_status": %{resp.status}V,
  "response_body_size": %{resp.body_bytes_written}V,
  "response_content_type": "%{json.escape(resp.http.Content-Type)}V",
  "fastly_server": "%{json.escape(server.identity)}V",
  "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V,
  "request_id": "%{json.escape(req.xid)}V",
  "signature": "%{json.escape(req.http.Signature)}V",
  "signature_input": "%{json.escape(req.http.Signature-Input)}V",
  "signature_agent": "%{json.escape(req.http.Signature-Agent)}V"
}
```

Mix the two as you like: the four minimal keys are required, every other key can be left out, empty or `(null)`. Keep `fastly_is_edge` whenever your service has a shield, otherwise each miss is counted twice. The unquoted keys (`timestamp`, `response_status`, `response_body_size`, `fastly_is_edge`) must always have a value, or the line is not valid JSON.

> **INFO**
> Fastly Compute services write the same JSON from code, one object per line, with the endpoint settings from Step 2.

## Step 3 — Verify

Fastly delivers logs in small batches, usually within a minute of the requests. Bot visits then appear in the Bot Analytics report a few minutes later. Requests from browsers are sent too; Ahrefs discards everything that is not a bot.

### Troubleshooting

Fastly does not show delivery errors for HTTPS endpoints in the control panel, so check the settings against the table above when nothing arrives. Ahrefs answers each batch with one of these:

| Response | Cause |
|---|---|
| `401` `missing authentication header` | The custom header name is misspelled or empty. |
| `403` `invalid authentication` | The custom header value does not match an Ahrefs project. Copy it again from Step 1. |
| `400` `record 0: …` | The log line format is **Classic** instead of **Blank**, the compression is not Gzip or none, or the log format is not valid JSON. |
| `400` `record N: Missing record field …` | Newline delimited: the log format lacks one of the required keys. |
| `400` `invalid body: not a JSON array of log records: …` | Array of JSON: the log line format is **Classic** instead of **Blank**, the log format lacks a required key, or it is not valid JSON. |
| `400` `body is not valid gzip` | Compression is **Gzip** in Fastly but the body could not be decompressed. |