There’s different ways to create a robots.txt file. You can create it from your:
- Content Management System
- Computer after which you upload it through your web server
Creating a robots.txt from your Content Management System
Oftentimes, modern Content Management Systems (CMS) have functionality to create and maintain your robots.txt file from within the CMS. If it doesn’t come with the CMS by default, you can often find plugins to help with this.
WordPress + Yoast SEO
For example, the Yoast SEO plugin on WordPress support creating and maintaining the robots.txt file:
WordPress + Rank Math
To create and maintain the robots.txt file in Rank Math SEO plugin:
WordPress + All in One SEO
And in All in One SEO Pack plugin, here’s how you create a robots.txt:
Magento 2
In Magento 2, your robots.txt file is located here:
On the screen that follows, scroll down to Search Engine Robots. There you can define your robots.txt content:
Shopware 5
Because Shopware 5 doesn’t come with a robot.txt editor out of the box, you’ll need to install a plugin or make adjustments to the code that generates the robots.txt.
Creating a robots.txt on your computer
If you’re not using a CMS, or the CMS doesn’t support creating a robots.txt you can always create a robots.txt yourself and manually upload it to your web server.
