"Indexed, though blocked by robots.txt": what does it mean and how to fix?
“Indexed, though blocked by robots.txt” indicates that Google indexed URLs even though they were blocked by your robots.txt file.
Google has marked these URLs as “Valid with warning” because they’re unsure whether you want to have these URLs indexed. In this article you'll learn how to fix this issue.
Here’s what this looks like in Google Search Console’s Index Coverage report, with the amount of URL impressions shown:
Double-check on URL level
You can double-check this by going to
Indexed, though blocked by robots.txt and inspect one of the URLs listed. Then under
Crawl it'll say "No: blocked by robots.txt" for the field
Crawl allowed and "Failed: Blocked by robots.txt" for the field
Submit your website, and find out right away!
So what happened?
Normally, Google wouldn’t have indexed these URLs but apparently they found links to them and deemed them important enough to be indexed.
It’s likely that the snippets that are shown are suboptimal, such as for instance:
How to fix “Indexed, though blocked by robots.txt”
Track every change you make, and make sure your robots.txt don't further hurt your site!
Indexed, though blocked by robots.txt fix for WordPress
The process to fixing this issue for WordPress sites is the same as described in the steps above, but here are some pointers to quickly find your robots.txt file in WordPress:
WordPress + Yoast SEO
If you’re using the Yoast SEO plugin, follow the steps below to adjust your robots.txt file:
WordPress + Rank Math
If you’re using the Rank Math SEO plugin, follow the steps below to adjust your robots.txt file:
WordPress + All in One SEO
If you’re using the All in One SEO plugin, follow the steps below to adjust your robots.txt file:
Indexed, though blocked by robots.txt fix for Shopify
Shopify doesn’t allow you to manage your robots.txt from their system, so you’re working with a default one that’s applied to all sites.
Perhaps you’ve seen the “Indexed, though blocked by robots.txt” message in Google Search Console or received a “New index coverage issue detected” email from Google about it. We recommended to always check out what URLs this concerns, because you don’t want to leave anything to chance in SEO.
Review the URLs, and see if any important URLs are blocked. If that’s the case, you’ve got two options which require some work, but do allow you to change your robots.txt file on Shopify:
Whether or not these options are worth it to you depends on the potential reward. If it’s sizable, look into implementing one of these options.
You can take the same approach on the Squarespace platform.
🤖 Why is Google showing this error for my pages?
Google found links to pages that aren't accessible to them due to robots.txt disallow directives. When Google deems these pages important enough, they'll index them.
🧐 How do you fix this error?
The short answer to that, is by making sure pages that you want Google to index should just be accessible to Google's crawlers. And pages that you don't want them to index, shouldn't be linked internally. The long answer is described in the section "How to fix 'Indexed, though blocked by robots.txt'" of this article.
🧾 Can I edit my robots.txt file on WordPress?
Popular SEO plugins such as Yoast, Rank Math and All in one SEO for example allow you to edit your robots.txt directly from the wp-admin panel.