On top of monitoring your website's pages and XML Sitemaps, ContentKing also monitors your robots.txt file. Your robots.txt file is checked every 5 minutes.
Working with robots.txt in ContentKing
ContentKing monitors your robots.txt file 24/7 and you're alerted in case it changes or becomes inaccessible.
ContentKing also stores the robots.txt versions historically, so you can compare the changes between the versions, see the response headers that ContentKing received, or download the robots.txt data!
Comparing different versions of robots.txt
ContentKing not only stores robots.txt versions historically - you can also compare the changes between them!
You noticed that the search engines haven't been crawling certain pages due to the robots.txt blocking them?
Easily compare different robots.txt versions that ContentKing indexed and see when was the undesired rule implemented:
Robots.txt and Alerts
Alerts in ContentKing notify you when any important changes happen on your website and allow you to take action before they hurt your bottom line.
ContentKing out of the box alerts you whenever any changes are made to the robots.txt file, or when it becomes inaccessible.
You can find more details about Alerts in ContentKing here: Alerts
Robots.txt and Issues
ContentKing audits the robots.txt file for the following SEO issues:
- Robots.txt is accessible
- Robots.txt is present on canonical variant only
- All directives are valid
- Contains no crawl-delay directives
- Syntax is valid
- All assets accessible to search engines
- XML sitemap reference is present
You can read more about issues that ContentKing audits here: Issues
See HTTP headers for any robots.txt version
Ever wondered what the exact response was for your robots.txt? Or are you seeing two alternating versions of your robots.txt and need to investigate?
Check the HTTP headers for any robots.txt version to see:
- the headers of the HTTP request that we sent
- the headers of the HTTP response that we received
Download your robots.txt data with one click
Need to show the response and HTTP headers to your colleagues or the technical team? Easily download your robots.txt data and forward it directly to them.
See when your robots.txt is visited by search engines in real-time
You implemented an important change to the robots.txt file and are wondering if search engines picked it up?
With our new Log File Analysis feature, you can know whether or not Google and Bing have seen each version directly in the robots.txt viewer.
In case you have any questions regarding the robots.txt monitoring in ContentKing that are not covered by our documentation, don’t hesitate to contact us!