Robots.txt file: Difference between revisions

Latest revision as of 11:59, 25 October 2022

About Robots

Web robots, also known as crawlers, web wanderers or spiders, are programs that traverse the web automatically. Among many uses, search engines use them to index the web content.

The robots.txt file implements the Robots Exclusion Protocol (REP), which allows the website administrator to define what parts of the site are off limits to specific robot user agents. Web administrators can Allow access to their web content and Disallow access to cgi, private and temporary directories, for example, if they do not want pages in those areas indexed.

Where to Place the robots.txt File

A standard robots.txt file is included in your Joomla root. The robots.txt file must reside in the root of the domain or subdomain and must be named robots.txt.

Joomla in a Subdirectory

A robots.txt file located in a subdirectory isn't valid. The bots only check for this file in the root of the domain. If the Joomla site is installed within a folder such as example.com/joomla/, the robots.txt file must be moved to the site root at example.com/robots.txt.

Note: The Joomla folder name must be prefixed to the disallowed path. For example, the Disallow rule for the /administrator/ folder must be changed to read Disallow: /joomla/administrator/

Joomla robots.txt Contents

This is the contents of a standard Joomla robots.txt:

User-agent: *
Disallow: /administrator/
Disallow: /api/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

Robot Exclusion

You can exclude directories or block robots from your site by adding a Disallow rule to the robots.txt file. For example, to prevent any robots from visiting the /tmp directory, add this rule:

Disallow: /tmp/

Syntax Checking

For syntax checking you can use a validator for robots.txt files. Try one of these:

Test your robots.txt with the robots.txt Tester at Google.
robots.txt Checker by Search Engine Promotion Help.

General Information

The Web Robots Pages The main Website for robots.txt.
A Standard for Robot Exclusion The original standard.
Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Robots.txt and Search Indexing

@@ Line 1: / Line 1: @@
-{{RightTOC}}
+<noinclude><languages /></noinclude>
-{{review}}
+<translate>
-Web Robots (Crawlers, Web Wanderers or Spiders) are programs that traverse the Web automatically. Among many uses, search engines use them to index the web content.
+== About Robots == <!--T:1-->
-Robots.txt implements the REP (Robots Exclusion Protocol), which allows the web site administrator to define what parts of the site are off-limits to specific robot user agent names. Web administrators can Allow access to their web content and Disallow access to cgi, private and temporary directories, for example, if they do not want pages in those areas indexed.
+Web robots, also known as crawlers, web wanderers or spiders, are programs that traverse the web automatically. Among many uses, search engines use them to index the web content.
-==Where to place my robots.txt file?==
+<!--T:2-->
-A standard robots.txt its included in your joomla root.
+The robots.txt file implements the [[wp:Robots exclusion standard|Robots Exclusion Protocol]] (REP), which allows the website administrator to define what parts of the site are off limits to specific robot user agents. Web administrators can Allow access to their web content and Disallow access to cgi, private and temporary directories, for example, if they do not want pages in those areas indexed.
-The robots.txt file must reside in the root of the domain and must be named "robots.txt".
-===Joomla in a subdomain===
+== Where to Place the ''robots.txt'' File == <!--T:3-->
-A robots.txt file located in a subdirectory isn't valid, as bots only check for this file in the root of the domain.
+A standard ''robots.txt'' file is included in your Joomla root. The ''robots.txt'' file must reside in the root of the domain or subdomain and must be named <code>robots.txt</code>.
-If the Joomla site is installed within a folder such as at e.g. www.example.com/joomla/ the robots.txt file MUST be moved to the site root at e.g. www.example.com/robots.txt .
-Note: The joomla folder name MUST be prefixed to the disallowed path, e.g. the Disallow rule for the /administrator/ folder MUST be changed to read Disallow: /joomla/administrator/
-==Joomla robots.txt contents==
+=== Joomla in a Subdirectory === <!--T:4-->
-This is the contents of a standard Joomla robots.txt
+A robots.txt file located in a subdirectory isn't valid. The bots only check for this file in the root of the domain.
+If the Joomla site is installed within a folder such as ''example.com/joomla/'', the ''robots.txt'' file '''must''' be moved to the site root at ''example.com/robots.txt''.
-<source lang="php">
+<!--T:5-->
+Note: The Joomla folder name '''must''' be prefixed to the disallowed path. For example, the Disallow rule for the <code>/administrator/</code> folder '''must''' be changed to read <code>Disallow: /joomla/administrator/</code>
+== Joomla ''robots.txt'' Contents == <!--T:6-->
+This is the contents of a [https://github.com/joomla/joomla-cms/blob/4.2-dev/robots.txt.dist standard Joomla robots.txt]:
+<!--T:7-->
+<syntaxhighlight lang="text">
 User-agent: *
 Disallow: /administrator/
+Disallow: /api/
+Disallow: /bin/
 Disallow: /cache/
 Disallow: /cli/
 Disallow: /components/
-Disallow: /images/
 Disallow: /includes/
 Disallow: /installation/
 Disallow: /language/
+Disallow: /layouts/
 Disallow: /libraries/
 Disallow: /logs/
-Disallow: /media/
 Disallow: /modules/
 Disallow: /plugins/
-Disallow: /templates/
 Disallow: /tmp/
-</source>
+</syntaxhighlight>
-==Robot Exclusion==
+== Robot Exclusion == <!--T:8-->
-You can exclude directories or block robots from your site adding Disallow rule to the robots.txt
+You can exclude directories or block robots from your site by adding a Disallow rule to the ''robots.txt'' file. For example, to prevent any robots from visiting the ''/tmp'' directory, add this rule:
-Infos:
+<!--T:9-->
-* [http://www.robotstxt.org/orig.html A Standard for Robot Exclusion]
+<syntaxhighlight lang="text">
-* [http://support.google.com/webmasters/bin/answer.py?hl=en-GB&answer=156449 Block or remove pages using a robots.txt file]
+Disallow: /tmp/
+</syntaxhighlight>
-==Syntax checking==
-For syntax checking you can use a validator for robots.txt files.
-Try one of these:
-* [http://tool.motoricerca.info/robots-checker.phtml Robots.txt Checker (by Motoricerca)]
-* [http://www.frobee.com/robots-txt-check Robots.txt Checker (by Frobee)]
-* [http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php robots.txt Checker (by Search Engine Promotion Help)]
-==Infos==
-For additional infos please read:
-===Joomla! Documentation===
-* [[Creating_a_Custom_404_Error_Page|Creating a Custom 404 Error Page]]
-* [[Custom_error_pages|Custom Error Pages]]
-* [[System_error_pages|System Error Pages]]
-* [[Making_your_site_Search_Engine_Friendly|Making your site Search Engine Friendly]]
-* [[Entering_search_engine_meta-data|Entering search engine meta-data]]
-===How to: Robots.txt and Joomla===
-* [http://www.kangainternet.com.au/joomla-seo-blog/joomla-google-the-robots.txt-file.html Joomla & Google Robots Text File]
-* [http://www.joomlablogger.net/seo/joomla-seo/how-to-set-up-the-robotstxt-file-in-joomla How to set up the robots.txt file in Joomla]
-===General informations===
+<!--T:10-->
-* [http://www.robotstxt.org/orig.html The Web Robots Pages]
+See also:
-* [https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag Robots meta tag and X-Robots-Tag HTTP header specifications]
+* [https://support.google.com/webmasters/topic/4598466?hl=en&ref_topic=9427949 Block access to your content] at Google's Help Center.
-* [http://www.searchtools.com/robots/robots-txt.html Robots.txt and Search Indexing]
-===Tools for Webmasters===
+== Syntax Checking == <!--T:11-->
-* [https://www.google.com/webmasters/tools Google Webmaster Tools]
+For syntax checking you can use a validator for ''robots.txt'' files. Try one of these:
+* [https://support.google.com/webmasters/answer/6062598 Test your ''robots.txt'' with the robots.txt Tester] at Google.
+* [http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php ''robots.txt'' Checker] by Search Engine Promotion Help.
-[[Category:Search Engine Optimization]]
+=== General Information === <!--T:12-->
+* [http://www.robotstxt.org/ The Web Robots Pages] The main Website for ''robots.txt''.
+* [http://www.robotstxt.org/orig.html A Standard for Robot Exclusion] The original standard.
+* [https://developers.google.com/search/docs/advanced/robots/robots_meta_tag Robots meta tag, data-nosnippet, and X-Robots-Tag specifications]
+* [https://searchtools.com/robots/robots-txt.html Robots.txt and Search Indexing]
+</translate>
+[[Category:Search Engine Optimisation]]
+[[Category:Content Management]]