Proper setup of Robots.txt for Joomla site

Proper setup of Robots.txt for Joomla websites plays a pivotal role in the site's SEO. Any error can result in pages being dropped from the Yandex and Google index, leading to decreased traffic. This file determines how search engines see the site.

For instance, using the default Robots.txt in Joomla will cause you to forget about indexing images hosted on it. Additionally, the search bot won't understand the website's design, certain content will be lost, and the bot's ability to navigate through menu links will be hindered. Original module designs make the website more user-friendly and colorful, but the default robots.txt file doesn't let search bots appreciate this.

Joomla developers are aware of this issue, and in versions from 3.3, the robots.txt file has seen significant enhancements. However, during a CMS upgrade, the previous version's file isn't updated automatically - a file is uploaded to the root directory with an .txt.dist extension, which needs editing. Moreover, changes specific to the website still need to be incorporated.

Making necessary changes to Robots.txt

The enhancement from the developers of the new Robots.txt version for Joomla involves removing three prohibitive directives from image, background, and media folders. To give search bots a complete view of the site, with all its unique features, this isn't enough. To understand this, you can compare in Google Search Console how a user sees the site versus a bot.

Upon inspection, it will be evident that videos, backgrounds of individual modules (from version 2.5, they can have unique designs) are inaccessible to the bot, and the site's markup isn't displayed. Moreover, site analysis will reveal that the search bot couldn't access the plugins folder, components, and template backgrounds (tmp).

To fix this, one needs to delete the corresponding directives or use a permitting directive, pointing to the desired component. Removing them would grant access to all plugins, which isn't always appropriate. Sometimes, plugins are installed to hide outgoing links, and showing this to the search bot might be frowned upon.

Additionally, it's essential to remove the ban on the modules folder since that's where modules with their own backgrounds and designs are installed, which the search bot won't see otherwise.

The same applies to the components folder, to grant the bot access to scripts, style files, and images on that path.

Example of a correct Robots.txt for Joomla:

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Allow: /media/*.css
Allow: /media/*.js
Allow: /media/*.png
Allow: /media/*.woff
Allow: /media/*.woff2
Allow: /media/*.ttf
Allow: /media/*.svg
Allow: /modules/*.css
Allow: /modules/*.js
Allow: /components/*.css
Allow: /components/*.js
Allow: /components/*.png
Allow: /plugins/*.css
Allow: /plugins/*.js
Allow: /plugins/*.png
Allow: /templates/*.css
Allow: /templates/*.js
Allow: /templates/*.woff
Allow: /templates/*.woff2
Allow: /templates/*.eot
Allow: /templates/*.ttf
Allow: /templates/*.otf
Allow: /templates/*.svg
Allow: /templates/*.png
Allow: /templates/*.jpg
Allow: /templates/*.gif
Host: Site_Domain

Besides, it's believed that microdata is integrated into Joomla. However, in reality, most users use commercial templates where the content templates are overwritten and differ significantly from the defaults. The path to them in Robots.txt needs to be added manually.

Proper Robots.txt setup for Joomla websites is quite simple and takes so little time. Yet, there are sites where this task is neglected for years, leading to a loss in traffic and an increased SEO promotion budget. To avoid all of this, one just needs to take these simple steps!