Expanded robots.txt monitoring with more granular change tracking

  • February 2, 2022
  • Veronika Martosova

ContentKing has always kept an eye on your website’s most important file: robots.txt.

Now we’re taking it a step further with our expanded robots.txt monitoring! Our expanded robots.txt monitoring makes it easier to navigate, view, and analyze your robots.txt files with:

  • Extensive change tracking of robots.txt
  • HTTP headers for any robots.txt versions
  • Ability to download full response
  • Log file analysis for robots.txt

And not to mention, we made it look pretty good too! Our new redesign makes comparing different robots.txt versions a breeze.

Travel back in time and see all versions of robots.txt

Quickly and easily see all changes that have happened in your robots.txt in one place.

Robots.txt in ContentKing with change tracking and option for a comparison between robots.txt versions.

See HTTP headers for any robots.txt version

Ever wondered what the exact response was for your robots.txt? Or are you seeing two alternating versions of your robots.txt and need to investigate?

Check the HTTP headers for any robots.txt version to see:

  • the headers of the HTTP request that we sent
  • the headers of the HTTP response that we received
Robots.txt in ContentKing showing HTTP headers for robots.txt.

Download your robots.txt data with one click

Need to show the response and HTTP headers to your colleagues or the technical team? Easily download your robots.txt data and forward it directly to them.

Robots.txt in ContentKing showing download button for robots.txt files.

See when your robots.txt is visited by search engines in real time

With our new Log File Analysis feature, you can know whether or not Google and Bing have seen each version directly in the robots.txt viewer.

Log File Analysis is available on our Pro and Enterprise plan. Haven’t checked out our new Log File Analysis feature yet? Learn more here.

Robots.txt in ContentKing showing that Google and Bing visited each version of robots.txt.

Try it out now

Don’t wait any longer! Check out these new improvements to robots.txt monitoring today.

Check your robots.txt now (opens in a new tab)

Want to learn more? Reach out to us if you have any questions.

Not using ContentKing yet? Do your website a favor and start your free trial right now!

Veronika Martosova

Veronika is ContentKing's Product Manager. She loves talking to our customers to understand their needs and then work with the development team to come up with the right solutions to them. She enjoys working on the intersection of design, development, marketing and business.