Academy

ContentKing's crawl speed depends on your website’s size, your webserver’s capacity and (optionally) the monitoring speed you set.

The relation between website size and crawl speed

The larger the website, the faster ContentKing needs to crawl your website in order to perform its monitoring duties. So if your website is only a couple of thousand pages, the crawl frequency will be very low. For example: if your website has 2,000 pages, ContentKing will request a page once every 10 seconds.

If your website has a lot of pages, ContentKing will crawl faster in order to keep up to date on your website’s changes as well as possible. For example: if your website has 20,000 pages, ContentKing will request a page every 3 seconds, and with 200,000 pages, ContentKing will request a page every second.

The relation between web server capacity and crawl speed

ContentKing continuously assesses the performance of your website so as to avoid overloading it. It does this through its throttling algorithm, which looks at the time it takes to get a response from the webserver and at the response itself.

Whenever the algorithm detects a problem, it automatically and immediately slows down crawling until the impact on the webserver has remedied.

Setting a different crawl speed

You can speed up or slow down ContentKing's crawling by setting a different monitoring speed in the website settings in ContentKing.

ContentKing monitoring speed

To go faster than the default monitoring speed you need to verify domain ownership by connecting with Google Search Console or Google Analytics.

Setting different crawl speeds at different times of the day

If your website sees significant higher loads due to traffic at peak times it's possible to configure ContentKing to slow down, or even pause, monitoring during these times by using the Peak / Off-peak monitoring speed feature:

ContentKing monitoring with different peak and off-peak speeds

Start your free 14-day trial

Get up and running in 20 seconds

Please enter a valid domain name (www.example.com).
  • No credit card required
  • No installation needed
  • No strings attached