Setting up the Fastly integration

Setting up Log File Analysis with the Fastly Real-Time Log Streaming (opens in a new tab) in ContentKing is very straightforward! Follow the steps outlined below to enable Log File Analysis in ContentKing, and set up the Fastly integration.

Setting up the Log File Analysis

The process of configuring the Log File Analysis consists of two phases:

1. Enabling the Log File Analysis feature and creating an AWS S3 bucket (opens in a new tab) in ContentKing.

2. Configuring the Fastly Real-Time Log Streaming in the Fastly UI.

1. Enabling the Log File Analysis feature and creating an AWS S3 bucket in ContentKing

If you navigate to Account and then Websites section in ContentKing, you can easily filter and see which of the websites you are monitoring in ContentKing are on Fastly:

From there, just follow the steps outlined below to enable Log File Analysis on the wanted website:

After this is done, follow the next steps to create an AWS S3 bucket in ContentKing:

2. Configuring the Fastly Real-Time Log Streaming

The Fastly Real-Time Log Streaming needs to be manually configured in the Fastly UI for every website.

The configuration of the Fastly Real-Time Log Streaming consists of these steps:

1. Open website settings screen in your Fastly account

Log in to your Fastly account (opens in a new tab) and click on the website (service) for which you want to enable the Log File Analysis feature

2. Click the Edit configuration button

In the service summary screen, click Edit configuration

3. Open the Logging endpoints screen and create a new endpoint

In the left-hand nav bar click on Logging, and then click Create endpoint

3. Select the Amazon S3 endpoint

In the Amazon S3 section click Create endpoint

4. Configure the S3 endpoint

In the S3 endpoint details screen add the following settings:

• Name: enter any name that would help easily identify the endpoint, e.g, "ContentKing LFA S3"
• Placement: Format Version Default (this is the default, no change needed)
• Log format: Paste the following format

{   
    "timestampMs": %{time.start.msec}V,
    "ip": "%{req.http.Fastly-Client-IP}V",
    "url": "%{if(req.is_ssl, "https", "http")}V://%v%U%q",
    "method": "%{json.escape(req.method)}V",
    "userAgent": "%{json.escape(req.http.User-Agent)}V",
    "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V"
  }

• Timestamp format: %Y-%m-%dT%H:%M:%S.000 (this is the default one, no change needed)
• Bucket name: copy-paste here the bucket name from the modal in ContentKing that was generated in the steps above.
• Access Method: User Credentials
• Access key: copy-paste here the Access key from the modal in ContentKing that was generated in the steps above.
• Secret key: copy-paste here the Access key from the modal in ContentKing that was generated in the steps above.
• Period: 60 seconds

5. Configure additional S3 settings

Click on the Advanced options and edit more properties:

• Path: / (this is default, no need to change this)
• Domain: copy-paste here the Domain from the modal in ContentKing that was generated in the steps above.
• PGP public key: keep this empty (this is default, no need to change this)
• Select a log line format: Blank (this is default, no need to change this)
• Compression: Gzip
• The rest of the settings: keep the default ones

6. Save the settings

Once done, click Update

Filter away non-search engine traffic

Setting up the response condition

To configure a response condition, so only search engine traffic is sent to the S3 bucket, follow the steps below:

1. Open Conditions

In the left-hand nav bar select Conditions

2. Create a condition

In the Condition screen, click Create condition

3. Configure the condition to filter away non-search traffic

In the Condition details screen fill in the following condition:

• Type: Response
• Name: any name, e.g. "Is Crawler"
• Apply if… Paste the exact condition below

req.http.user-agent ~ "bingbot" || req.http.user-agent ~ "googlebot"

4. Save the condition

To save the condition, click Save

Attaching the condition to the needed endpoint

To attach the condition to the created endpoint follow these steps:

1. Open the Logging endpoints screen and attach the condition

Open the Logging endpoints screen by clicking on Logging in the left-hand nav bar, and click Attach condition for the endpoint you've just created

2. Select the needed condition to attach

In the Add a condition screen select the condition you've just set up

3. Save all changes

To save all changes, just click Activate

If everything has been configured correctly, you should see a success message

Reinstalling the Fastly Real-Time Log Streaming

If you want to change the AWS S3 bucket region, or reinstall the Fastly Real-Time Log Steaming, you can do it in the following way:

Disabling Log File Analysis

Same as with enabling Log File Analysis, you need to disable the feature in ContentKing and then remove the Fastly Real-Time Log Streaming in the Fastly UI.

Disabling Log File Analysis

Once this is done, ContentKing will automatically disable Fastly Real-Time Log Streaming access to the AWS S3 bucket where the data was streamed.

Removing the Fastly Real-Time Log Streaming

If you have disabled the Log File Analysis feature, you still need to remove the Fastly Real-Time Log Streaming in your Fastly account.

This needs to be done manually, as ContentKing doesn't have access to your Fastly account.

Frequently asked Security related questions

For the most common security-related questions about ContentKing's Log File Analysis, refer to the FAQ section in the Log File Analysis support article

Need help?

In case you have any questions regarding the Fastly integration that are not covered by our documentation, don’t hesitate to contact us!