Robots.txt monitoring

On top of monitoring your website's pages and XML Sitemaps, Conductor Website Monitoring also monitors your robots.txt file. Your robots.txt file is checked every 5 minutes.

Read more about robots.txt

• Robots.txt guide for SEO (opens in a new tab)

Working with robots.txt in Conductor Website Monitoring

Conductor Website Monitoring monitors your robots.txt file 24/7 and you're alerted in case it changes or becomes inaccessible.

Conductor Website Monitoring also stores the robots.txt versions historically, so you can compare the changes between the versions, see the response headers that Conductor Website Monitoring received, or download the robots.txt data!

Comparing different versions of robots.txt

Conductor Website Monitoring not only stores robots.txt versions historically - you can also compare the changes between them!

You noticed that the search engines haven't been crawling certain pages due to the robots.txt blocking them?

Easily compare different robots.txt versions that Conductor Website Monitoring indexed and see when was the undesired rule implemented:

Robots.txt and Alerts

Alerts in Conductor Website Monitoring notify you when any important changes happen on your website and allow you to take action before they hurt your bottom line.

Conductor Website Monitoring out of the box alerts you whenever any changes are made to the robots.txt file, or when it becomes inaccessible.

You can find more details about Alerts in Conductor Website Monitoring here: Alerts

Robots.txt and Issues

Conductor Website Monitoring audits the robots.txt file for the following SEO issues:

• Robots.txt is accessible
• Robots.txt is present on canonical variant only
• All directives are valid
• Contains no crawl-delay directives
• Syntax is valid
• All assets accessible to search engines
• XML sitemap reference is present

You can read more about issues that Conductor Website Monitoring audits here: Issues

See HTTP headers for any robots.txt version

Ever wondered what the exact response was for your robots.txt? Or are you seeing two alternating versions of your robots.txt and need to investigate?

Check the HTTP headers for any robots.txt version to see:

• the headers of the HTTP request that we sent
• the headers of the HTTP response that we received

Download your robots.txt data with one click

Need to show the response and HTTP headers to your colleagues or the technical team? Easily download your robots.txt data and forward it directly to them.

See when your robots.txt is visited by search engines in real-time

You implemented an important change to the robots.txt file and are wondering if search engines picked it up?

With our new Log File Analysis feature, you can know whether or not Google and Bing have seen each version directly in the robots.txt viewer.

Need help?

In case you have any questions regarding the robots.txt monitoring in Conductor Website Monitoring that are not covered by our documentation, don’t hesitate to contact us!