Academy

Should I block SEMrush in my robots.txt?

The answer is: it depends.

If SEMrush is used to crawl your website, then it doesn’t make sense to block them.

If you’re not using SEMrush, and you want to preserve bandwidth or prevent SEMrush from indexing backlinks then you can block their bot.

How to block SEMrush through your robots.txt?

Block SEMrush bots entirely

This example is pretty straight forward, it just blocks SEMrush’s bots entirely:

User-agent: SemrushBot
Disallow: /

Tell SemrushBot to go easy on your site

Alternatively, you could tell SemrushBot to crawl your site slowly if they’re taking up too much of your site’s resources. Here’s how to tell them to crawl one URL per minute:

User-agent: SemrushBot
Crawl-delay: 60

And say you only want to block their backlink audit tool, but allow their other tools to access the site you can put this in your robots.txt:

User-agent: SemrushBot-BA
Disallow: /

How long does it take before SEMrush obeys my robots.txt?

According to SEMrush, it can take them up to one hour or 100 requests to re-check your robots.txt file and process its contents.

Further reading

If you want to read more about this topic, check out the SEMrushbot documentation.

ContentKing Academy

Read the full Academy article to learn everything about Robots.txt

Start your free 14-day trial

Get up and running in 20 seconds

Please enter a valid domain name (www.example.com).
  • No credit card required
  • No installation needed
  • No strings attached