Academy

How do I create a robots.txt?

There’s different ways to create a robots.txt file. You can create it from your:

  1. Content Management System
  2. Computer after which you upload it through your web server

Creating a robots.txt from your Content Management System

Oftentimes, modern Content Management Systems (CMS) have functionality to create and maintain your robots.txt file from within the CMS. If it doesn’t come with the CMS by default, you can often find plugins to help with this.

WordPress + Yoast SEO

For example, the Yoast SEO plugin on WordPress support creating and maintaining the robots.txt file:

  1. Log into your wp-admin section.
  2. In the sidebar, go to Yoast SEO plugin > Tools.
  3. Go to File editor.

WordPress + Rank Math

To create and maintain the robots.txt file in Rank Math SEO plugin:

  1. Log into your wp-admin section.
  2. In the sidebar, go to Rank Math > General Settings.
  3. Go to Edit robots.txt.

WordPress + All in One SEO

And in All in One SEO Pack plugin, here’s how you create a robots.txt:

  1. Log into your wp-admin section.
  2. In the sidebar, go to All in One SEO > Robots.txt.

Magento 2

In Magento 2, your robots.txt file is located here:

  1. Content > Configuration (Under Design).
  2. There, edit the settings for Main Website instead of the Default Store View, because the latter won’t allow you to change the robots.txt file.
Screenshot of the Design Configuration screen in Magento 2
Navigating to the robots.txt settings in Magento 2.

On the screen that follows, scroll down to Search Engine Robots. There you can define your robots.txt content:

Screenshot of the robots.txt instructions field in Magento 2
Adjusting the robots.txt instructions in Magento 2.

Shopware 5

Because Shopware 5 doesn’t come with a robot.txt editor out of the box, you’ll need to install a plugin or make adjustments to the code that generates the robots.txt.

Creating a robots.txt on your computer

If you’re not using a CMS, or the CMS doesn’t support creating a robots.txt you can always create a robots.txt yourself and manually upload it to your web server.

  1. Open an editor like Notepad on Windows, or textEdit on Mac OS X.
  2. Create a new file.
  3. Copy-paste one of the example robots.txt files.
  4. Adjust the contents as needed.
  5. Save it with the filename robots.txt.
  6. Upload it to your web server.
ContentKing Academy

Read the full Academy article to learn everything about Robots.txt

Start your free 14-day trial

Get up and running in 20 seconds

Please enter a valid domain name (www.example.com).
  • No credit card required
  • No installation needed
  • No strings attached