The crawl-delay directive is an unofficial directive meant to communicate to crawlers to slow down crrawling in order not to overload the web server.
Some search engines don’t support the crawl-delay directive, and the way the crawl-delay directive is interpreted varies across search engines.
How does Google interpret crawl-delay: 10?
Google doesn’t support the crawl-delay directive, so her crawlers will just ignore it.
If you want to ask Google to crawl slower, you need to set the
Crawl rate in Google Search Console:
This is what that looks like in Google Search Console:
Bing and Yahoo and crawl-delay: 10
Bing and Yahoo support the crawl-delay directive, and in case of crawl-delay: 10 they’ll divide a day into 10 second windows and within each window they’ll crawl a maximum of one page.
Read more about Bing’s interpretation of the crawl-delay directive.
Yandex and crawl-delay: 10
Yandex supports the crawl-delay directive, and in case of crawl-delay: 10 they’ll wait a minimum of 10 seconds before requesting another URL.
While Yandex supports this directive, they recommend using Yandex.Webmaster — their version of Google Search Console to define the crawl speed settings.
Read more about Yandex’ interpretation of the crawl-delay directive.
Baidu and crawl-delay: 10
Baidu does not support the crawl-delay directive, so — similar to Google — they’ll ignore the crawl-delay directive. You can define your prefered crawl frequency through Baidu Webmaster Tools.
