How to Configure Magento 2 Robots.txt?

Are you looking to configure Magento 2 robots.txt file?

The Robots.txt file gives instructions to the bots on what pages you want to index and what pages you want to leave alone.

It is important to configure it properly. 

In this tutorial, we will see how to configure robots.txt file in Magento 2 without using the Magento 2 SEO extensions.

What is Robots.txt file?

A website is made of hundreds of web pages. A Magento store typically contains thousands of pages. You do not want all the pages to be indexed by Google or other search engines.

Indexing means getting the page in the Google database. So when someone searches using a query related to the webpage, Google can show the page to the user.

Indexing is done through crawlers – tiny coded robots that explore the web all the time. They come to the website, understand the webpage and categorize it accordingly.

But imagine having all the website pages, including tags, categories, product variations, internal pages, admin pages and media URLs on Google. This is not wise in terms of SEO.

That’s why we limit the crawlers and gave them a right direction using the robot.txt file.

It is a small file to be added to the website header, so when the crawlers arrive at the website, they know what pages are allowed to be crawled.

Robots.txt file is a sophisticated solution to prevent over-indexing and ruining the store’s reputation.

How to Configure Magento 2 Robots.txt?

The Good thing is, you do not need to install the Magento extension to add the robots.txt file to the store. It can be done through inbuilt settings.

Follow the steps to configure the Magento 2 Robots.txt file:

1. Navigate to Content > Design > Configuration

It will open the list of all store views.

2. Select the Store (by clicking Edit) you want to configure the default Magento robots.txt file for

On the next page, you will find the store configuration.

3. Scroll down to find the Search Engine Robots section

4. Choose the default robots you want to configure for the website.

Let’s understand your options:

• INDEX, FOLLOW: Web crawlers will index the website and follow the links on the page to move to the next page.
• NOINDEX, FOLLOW: Web crawlers will not index the page but will follow the link on the page to move to the next page.
• INDEX, NOFOLLOW: Web crawlers will index the page but will not follow the link on the page and do not move to the next page.
• NOINDEX, NOFOLLOW: Neither the web crawlers will index the page, nor they will follow the link to move to the next page.

5. Next is Custom Instructions for Robots.

If you want to follow or noindex specific sections of the Magento store, you can leave the instructions here.

6. Click on the Save button to save the configuration.

You may have to flush the Magento cache to update the setting.

If you want to remove the custom instruction, click on the Reset to Defaults button. It will clear the custom instruction.

Magento 2 Robots.txt example

For example:

# Google Image Crawler Setup
User-agent: Googlebot-Image
Disallow:

# Crawlers Setup
User-agent: *

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

Note: Above Robots.txt file is only for the example.

The formula looks like this:

Allow Full Access

User-agent:*
Disallow:

Disallow Access to All Folders

User-agent:*
Disallow: /

Default Instructions

Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=

Restrict Checkout and Customer account

Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login

Restrict Catalog Search Pages

Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/

Disallow URL Filter Searches

Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /?mode

Restrict Common files

Disallow: /composer.json
Disallow: /composer.lock
Disallow: /CONTRIBUTING.md
Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow: /COPYING.txt
Disallow: /Gruntfile.js
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /nginx.conf.sample
Disallow: /package.json
Disallow: /php.ini.sample
Disallow: /RELEASE_NOTES.txt

Restrict Common Folders

Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/

Restrict Technical Magento Files

Disallow: / api.php
Disallow: / cron.php
Disallow: / cron.sh
Disallow: / error_log
Disallow: / get.php
Disallow: / install.php
Disallow: / LICENSE.html
Disallow: / LICENSE.txt
Disallow: / LICENSE_AFL.txt
Disallow: / README.txt
Disallow: / RELEASE_NOTES.txt

Restrict CMS Directories

Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/

FAQ

Is the robots.txt file work?

Yes. Robots.txt file is the best way to control the direction of bots. However, it is not a foolproof method, as it only handles the direction of the bots through your website. What if the crawlers found a different path to crawl the pages you were hiding?

How to find my robots.txt file?

Enter the store URL in your browser’s address bar and add /robots.txt to it. It will show the existing robots.txt file. For example, check Nike’s robots file: nike.com/robots.txt

Where can I test the robots.txt file?

You can check and verify the robots.txt file on the free Google Webmaster Tool. Add your robots.txt file and submit an URL to see if the crawlers are allowed to crawl the said URL.

Wrap Up

Search engine optimization is critical for the success of Magento stores. SEO is incomplete without the Magento 2 robots.txt file.

In this quick tutorial, we learn how to configure Magento 2 robots.txt file. If you encounter any issues, feel free to leave a comment.