What does crawl-delay: 10 mean in robots.txt?

The crawl-delay directive is an unofficial directive meant to communicate to crawlers to slow down crrawling in order not to overload the web server.

Some search engines don’t support the crawl-delay directive, and the way the crawl-delay directive is interpreted varies across search engines.

How does Google interpret crawl-delay: 10?

Google doesn’t support the crawl-delay directive, so her crawlers will just ignore it.

If you want to ask Google to crawl slower, you need to set the Crawl rate in Google Search Console:

This is what that looks like in Google Search Console:

Bing and Yahoo and crawl-delay: 10

Bing and Yahoo support the crawl-delay directive, and in case of crawl-delay: 10 they’ll divide a day into 10 second windows and within each window they’ll crawl a maximum of one page.

Read more about Bing’s interpretation of the crawl-delay directive (opens in a new tab).

Yandex and crawl-delay: 10

Yandex supports the crawl-delay directive, and in case of crawl-delay: 10 they’ll wait a minimum of 10 seconds before requesting another URL.

While Yandex supports this directive, they recommend using Yandex.Webmaster — their version of Google Search Console to define the crawl speed settings (opens in a new tab).

Read more about Yandex’ interpretation of the crawl-delay directive (opens in a new tab).

Baidu and crawl-delay: 10

Baidu does not support (opens in a new tab) the crawl-delay directive, so — similar to Google — they’ll ignore the crawl-delay directive. You can define your prefered crawl frequency through Baidu Webmaster Tools.