Docs
Szukaj w dokumentacji...⌘K

Azure Blob Storage

If your CDN can deliver its access logs to an Azure Blob Storage container, Ahrefs can pull them from there. You keep the storage account and the delivery in your own Azure subscription and give Ahrefs read access to the container: a SAS token for it, or an access key of the storage account. Ahrefs checks the container every few minutes and downloads the new log files. This works with Azure Front Door, Cloudflare Logpush, Akamai DataStream 2 and Fastly, as long as the logs are written as newline-delimited JSON.

For Azure Front Door, a diagnostic setting writes the access log to a storage account and you give Ahrefs access to its container. For other CDNs, use this method when you already collect their logs in Azure; otherwise the direct integrations (Cloudflare Logpush, Fastly) need no storage account key.

Prerequisites

  • A storage account in your Azure subscription, with a container that receives the access logs of the domain of your Ahrefs project.
  • Permission to read the storage account's access keys (the Storage Account Key Operator Service Role or an owner or contributor role on the storage account). Creating a SAS token needs them too.
  • The storage account name and the container name. Both are shown in the Azure portal under Storage accounts and the account's Data storage → Containers.
  • The logs must be newline-delimited JSON (one JSON object per line), plain or gzip-compressed, in one of the formats below. Parquet and tab-separated formats are not read.

warning

A SAS token for the container reaches only that container, and only reads it. A storage account key gives full access to everything in the account: if you use one, use a storage account that holds only these logs.

Step 1: Deliver the logs to the container

Configure the delivery in your CDN and note the storage account name, the container and the key prefix the files land under. Use a prefix that is only used for these logs, and a date-partitioned layout when the CDN offers one: Ahrefs lists the whole prefix on every poll, so it should stay small. A lifecycle management rule that deletes blobs after a few days keeps it that way; Ahrefs only needs each file once.

Log format in AhrefsCDN setup
Azure Front DoorOn the Front Door profile, go to Monitoring → Diagnostic settings → Add diagnostic setting. Select the FrontDoorAccessLog category and Archive to a storage account, and pick the storage account. Azure writes to the container insights-logs-frontdooraccesslog, one file per hour (.../PT1H.json) or per five minutes (.../PT5M.json). Enter that container name in Ahrefs and leave the key prefix empty.
Cloudflare LogpushA Logpush job for the http_requests dataset with a Microsoft Azure destination, output type NDJSON. Select the same fields as for the direct Logpush integration.
Akamai DataStreamA DataStream 2 stream with an Azure Storage destination, log format JSON.
FastlyA Microsoft Azure Blob Storage logging endpoint on your Fastly service with the log format from the Fastly guide, one JSON object per line, gzip, and Select a log line format set to Blank under Advanced options. The default, Classic, puts a syslog header before every line, and Ahrefs skips those files.

When you add the container, choose whether Ahrefs also imports the log files already in it. If you don't, Ahrefs starts with the files written after you add it, plus up to six hours before, to catch late deliveries.

info

Front Door keeps appending to each file until its hour, or its five minutes, ends. Ahrefs reads the lines added since its last read on every poll, so bot visits arrive a few minutes after Front Door logs them.

Step 2: Give Ahrefs access to the container

With the Azure CLI, a new storage account that holds only these logs, and an Azure Front Door profile writing to it, are set up like this. For another CDN, skip the diagnostic setting and point the CDN's delivery at the storage account instead.

# a storage account for the logs: 3 to 24 lowercase letters and digits
az storage account create -n <storage-account> -g <resource-group> -l <region> --sku Standard_LRS --kind StorageV2

# Front Door: archive its access log to that account
az monitor diagnostic-settings create --name <setting-name> \
  --resource "$(az afd profile show -g <resource-group> --profile-name <front-door-profile> --query id -o tsv)" \
  --storage-account "$(az storage account show -n <storage-account> -g <resource-group> --query id -o tsv)" \
  --logs '[{"category":"FrontDoorAccessLog","enabled":true}]'

Then create one of the two credentials below.

A SAS token for the container

The token allows Ahrefs to list the container and read its files, and nothing else. Tie it to a stored access policy: deleting the policy revokes the token, and changing the policy's expiry extends it, without touching your account keys.

  1. In the Azure portal, open the storage account, then Data storage → Containers and the log container.
  2. Under Settings → Access policy, click Add policy. Give it a name, select the Read and List permissions, set an expiry far enough ahead, and save.
  3. Under Settings → Shared access tokens, select Account key as the signing method and that policy as the stored access policy, keep HTTPS only, and click Generate SAS token and URL.
  4. Copy the Blob SAS URL, or the Blob SAS token.

The token is the part of the URL after the ?. Ahrefs accepts either:

https://<storage-account>.blob.core.windows.net/<container>?<token>
<token>

With the Azure CLI:

az storage container policy create --account-name <storage-account> --auth-mode key \
  -c <container> --name <policy-name> --permissions rl --expiry <expiry>

az storage container generate-sas --account-name <storage-account> --auth-mode key \
  -n <container> --policy-name <policy-name> --https-only -o tsv

For Front Door, the container exists only once the first log is written, so create the policy and the token after some traffic has gone through.

When the policy expires, polling stops with an access denied error. Extend the policy's expiry and polling resumes, with nothing to change in Ahrefs.

The storage account key

  1. In the Azure portal, open the storage account and go to Security + networking → Access keys.
  2. Click Show next to key1 and copy the Key. Don't copy the connection string.

With the Azure CLI:

az storage account keys list -n <storage-account> -g <resource-group> --query "[0].value" -o tsv

If you rotate the key later, polling stops with an access denied error: remove the container in Ahrefs and add it again with the new key.

Step 3: Add the container in Ahrefs

  1. In Ahrefs, go to Project Settings → Bot Analytics for your project and select Azure Blob Storage.
  2. Enter the container name, the storage account name, the key prefix, and pick the log format matching your CDN.
  3. Under Give Ahrefs access with, choose SAS token or Storage account key and paste it. A pasted SAS URL also fills in the storage account and the container.
  4. Leave Also import the log files already in the container checked to import its history, or uncheck it to start from now. Ahrefs reads up to 100,000 files under the prefix.
  5. Click Add container. Ahrefs lists the prefix once to check the access before saving; if that fails, the error tells you whether the container, the account or the key is wrong. Otherwise Ahrefs opens the Bot Analytics report.

You can add several containers or prefixes to the same project, for example one per CDN.

Step 4: Verify

Ahrefs polls each container about every five minutes and shows the last successful poll, or the last error, next to the container in the Bot Analytics settings. Most CDNs deliver log files with a delay of a few minutes, so the first bot visits appear in the Bot Analytics report shortly after the first file lands in the container.

Troubleshooting

MessageCause
access denied: check the SAS token, that it allows read and list, and its expiryThe token is wrong, lacks the Read or List permission, or expired, or its stored access policy was deleted or expired.
access denied: check the storage account keyThe key is wrong or was rotated. Copy the Key from Access keys, not the connection string.
access denied: check the storage account key, and that the account allows access from the internetBesides a wrong key, the storage account's firewall blocks public network access, or key access is disabled on the account.
storage account not found: check the storage account nameThe storage account name is misspelled, or the account was deleted.
container not found: check the container name and the storage account nameThe container name is misspelled, or the container is in another storage account. For Front Door, the container appears only after the first log is written.

If the prefix holds more than 100,000 blobs, Ahrefs reads only the first 100,000 it lists, without an error. Narrow the prefix or add a lifecycle management rule.

If the last poll succeeds but no bot visits appear after the CDN's delivery delay, the files are probably in another format. Ahrefs skips files that aren't newline-delimited JSON (tab-separated, Parquet, Fastly's Classic line format) and gzip files larger than 256 MB, without an error. Download one blob and check that each line starts with {.

Files that can't be read are skipped once and never retried. Everything else is retried with a growing delay after a failure.

Disabling

  1. In Ahrefs, in Project Settings → Bot Analytics, click Remove next to the container. Ahrefs stops polling it at once and deletes its copy of the token or key.
  2. In Azure, delete the container's stored access policy (SAS token) or rotate the storage account key, and disable the log delivery in your CDN if nobody else needs it.

info

Ahrefs reads each line of a log file once: the same lines are not imported twice even after a restart, and late deliveries are picked up for six hours after the newest file already imported.