Academy

What is a robot.txt file?

There’s no such thing as a robot.txt file. What you’re looking for is the robots.txt file.

Don’t worry about the typo, you’re in good company: loads and loads of people search for robot.txt instead of robots.txt.

So what is this robots.txt?

A robots.txt file explains the rules of engagement for your website to crawlers. Search engines rely on crawlers to gather information for them, and they use the robots.txt file as well to determine what parts of the site they can access, and what parts they need to stay out of.

The robots.txt file tells search engines what URLs not to access.
The robots.txt file tells search engines what URLs not to access.
ContentKing Academy

Read the full Academy article to learn everything about Robots.txt

Start your free 14-day trial

Get up and running in 20 seconds

Please enter a valid domain name (www.example.com).
  • No credit card required
  • No installation needed
  • No strings attached