How do I find my robots.txt file?

Last updated: August 6, 2021

Finding your robots.txt file on the front-end

Crawlers will always look for your robots.txt file in the root of your website, so for example: https://www.contentkingapp.com/robots.txt. Navigate to your domain, and just add "/robots.txt".

If nothing comes up, you don't have a robots.txt file yet. Don't worry, we explain here how to create a robots.txt file.

In case you want to update your robots.txt file: keep on reading.

Finding your robots.txt file on the back-end

If you're using a Content Management System (CMS), you may be able to manage it in there.

Your robots.txt in WordPress

We've explained where to find your robots.txt file in WordPress in this article covering the most popular plugins Yoast SEO, Rank Math and All in One SEO.

If you're working on a WordPress website that hasn't gone live yet, and you find the following contents in your robots.txt:

User-agent: *
Disallow: /

then check your settings under: Settings > Reading and look for Search Engine Visibility.

If the box Discourage search engines from indexing this site is checked, WordPress will generate a virtual robots.txt preventing search engines from accessing the site.

Found your robots.txt file? Is it limiting your SEO performance?

Audit your robots.txt and find out right away!

Magento 2

In Magento 2, your robots.txt file is located here:

Content > Configuration (Under Design).
There, edit the settings for Main Website instead of the Default Store View, because the latter won’t allow you to change the robots.txt file.

Screenshot of the Design Configuration screen in Magento 2 — Navigating to the robots.txt settings in Magento 2.

On the screen that follows, scroll down to Search Engine Robots. There you can define your robots.txt content:

Screenshot of the robots.txt instructions field in Magento 2 — Adjusting the robots.txt instructions in Magento 2.

Shopware

In Shopware, it's not possible to adjust your robots.txt file with the default installation.

You'll need to get a plugin (opens in a new tab) or make adjustments to the code (opens in a new tab) that generates the robots.txt.

Read the full Academy article to learn everything about Robots.txt

Name:	`viewedOuibounceModal`
Provider:	ContentKing
Purpose:	Store state of having closed the exit intent modal.
Expiry:	1 month
Type:	http_cookie

Name:	`nette-samesite`
Provider:	ContentKing
Purpose:	Communication with account service.
Expiry:	Session
Type:	http_cookie

Name:	`session_timestamp`
Provider:	ContentKing
Purpose:	Utility for tracking of session start time.
Expiry:	Session
Type:	SessionStorage

Name:	`what-intent`
Provider:	ContentKing
Purpose:	Utility for tracking the current input method.
Expiry:	Session
Type:	SessionStorage

Name:	`what-input`
Provider:	ContentKing
Purpose:	Utility for tracking the current input method.
Expiry:	Session
Type:	SessionStorage