{"id":747,"date":"2023-06-04T11:47:26","date_gmt":"2023-06-04T08:47:26","guid":{"rendered":"https:\/\/presta.site\/blog\/?p=747"},"modified":"2023-06-04T11:47:26","modified_gmt":"2023-06-04T08:47:26","slug":"a-simple-guide-to-configuring-robots-txt-in-prestashop","status":"publish","type":"post","link":"https:\/\/presta.site\/blog\/en\/a-simple-guide-to-configuring-robots-txt-in-prestashop\/","title":{"rendered":"A Simple Guide to Configuring Robots.txt in PrestaShop"},"content":{"rendered":"<p>Robots.txt is a small, but powerful file that tells search engine bots how to interact with your website. It&#8217;s like a guidebook, showing them which pages to visit (index) and which ones to skip (ignore). A well-set-up Robots.txt can help your website&#8217;s SEO. Actually it&#8217;s not just for PrestaShop, it can be found on any site. But in this post we will discuss how it works in PrestaShop.<\/p>\n<p><!--more--><\/p>\n<h2>Default Robots.txt in PrestaShop<\/h2>\n<p>Luckily for PrestaShop users, a robots.txt file is created automatically during the installation process. It&#8217;s set up for the needs of a typical eCommerce site. Most of the time, you won&#8217;t need to touch this file unless you have a specific reason.<\/p>\n<p><a href=\"http:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_11.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full wp-image-752\" src=\"http:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_11.png\" alt=\"\" width=\"737\" height=\"436\" srcset=\"https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_11.png 737w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_11-300x177.png 300w\" sizes=\"(max-width: 737px) 100vw, 737px\" \/><\/a><\/p>\n<h2>When You Need to Edit Robots.txt and When You Don\u2019t<\/h2>\n<p class=\"\">There are a few reasons why you might <strong>want to edit<\/strong> your Robots.txt file. Here are some examples:<\/p>\n<ul>\n<li>If you have two or more pages with the same content (<strong>duplicates<\/strong>), you might want to tell bots to ignore one to avoid a penalty from search engines.<\/li>\n<li>If you&#8217;ve added new pages or sections to your site, you might want to <strong>control<\/strong> how visible they are to search engines.<\/li>\n<li>Maybe you have pages or files you&#8217;d rather keep <strong>private\u00a0<\/strong>(however, keep in mind that instructions in Robots.txt are only <strong>recommendations<\/strong> for search engines)<\/li>\n<\/ul>\n<p>But remember, if you&#8217;re just using PrestaShop&#8217;s default functions and your SEO is working fine, there&#8217;s <strong>no need<\/strong> to change your Robots.txt.<\/p>\n<h2>Accessing the Robots.txt File<\/h2>\n<p>Your robots.txt file lives <strong>in the root directory<\/strong> of your PrestaShop site. You can use an FTP client or your hosting file manager to get there <strong>to edit it<\/strong>.<\/p>\n<p><strong>To view<\/strong> the file, just type &#171;https:\/\/YourSite.com\/robots.txt&#187; (your shop URL + &#171;robots.txt&#187;) into your browser&#8217;s address bar.<\/p>\n<h2>Generating Robots.txt in PrestaShop<\/h2>\n<p>In PrestaShop, you can re-generate Robots.txt file right from the Back Office. Here&#8217;s how to do it:<\/p>\n<ol>\n<li>Go to the &#171;<strong>Shop Parameters &gt;&gt; Traffic &amp; SEO<\/strong>&#187; page.<\/li>\n<li>Find the &#171;<strong>Robots file generation<\/strong>&#187; at the bottom.<\/li>\n<li>Click the &#171;<strong>Generate robots.txt file<\/strong>&#187; button.<\/li>\n<\/ol>\n<p><a href=\"http:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-large wp-image-753\" src=\"http:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-1024x205.png\" alt=\"\" width=\"660\" height=\"132\" srcset=\"https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-1024x205.png 1024w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-300x60.png 300w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-768x154.png 768w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1.png 1354w\" sizes=\"(max-width: 660px) 100vw, 660px\" \/><\/a><\/p>\n<p>That&#8217;s it, PrestaShop will generate a new robots.txt file. It&#8217;s useful if your robots.txt file is missing for some reason or if you edited the file and want to revert the changes.<\/p>\n<h2>Customizing Robots.txt<\/h2>\n<p>The Robots.txt file is written using a very simple text syntax, making it easy to understand and edit. Each rule in a Robots.txt file consists of two parts: a <strong>user-agent line<\/strong> and at least one <strong>Disallow line<\/strong>.<\/p>\n<h3>Here are some quick examples:<\/h3>\n<p><strong>Block everything:<\/strong><\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/><\/div><\/td><td><div class=\"text codecolorer\">User-agent: *<br \/>\nDisallow: \/<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p><strong>Block only Googlebot:<\/strong><\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/><\/div><\/td><td><div class=\"text codecolorer\">User-agent: Googlebot<br \/>\nDisallow: \/<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p><strong>Block specific directory:<\/strong><\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/><\/div><\/td><td><div class=\"text codecolorer\">User-agent: *<br \/>\nDisallow: \/private<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p><strong>Block specific directory except one URL:<\/strong><\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/><\/div><\/td><td><div class=\"text codecolorer\">User-agent: *<br \/>\nDisallow: \/private<br \/>\nAllow: \/private\/public-page<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<h2>Robots.txt syntax<\/h2>\n<h3><strong class=\"\">User-agent:<\/strong><\/h3>\n<p>This specifies the search engine bot that the rule will apply to. If you want to target all bots, you can use an asterisk (*). For example:<\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">User-agent: *<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>This rule applies to all bots.<\/p>\n<h3><strong>Disallow:<\/strong><\/h3>\n<p>This is followed by the relative URL path that you want to block from the bots. For example:<\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">Disallow: \/private<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>This rule tells the bot not to crawl or index the &#171;private&#187; directory.<\/p>\n<h3><strong class=\"\">Allow:<\/strong><\/h3>\n<p>This is used when you want to override a Disallow rule for a specific page or directory within a disallowed directory. For example:<\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">Allow: \/private\/public-page<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<h3>Specific bot<\/h3>\n<p>If you want to block a specific bot, you would specify that bot&#8217;s name in the User-agent line. For example, if you wanted to block Google&#8217;s bot (Googlebot) from indexing a specific directory, you could write:<\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">User-agent: Googlebot<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>And then the code for blocking a directory.<\/p>\n<h3>Sitemap<\/h3>\n<p>You can also provide a link to your sitemap in Robots.txt file:<\/p>\n<div class=\"codecolorer-container text blackboard\" style=\"overflow:auto;white-space:nowrap;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">Sitemap: https:\/\/example.com\/sitemap.xml<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<h2>Testing Robots.txt<\/h2>\n<p>After you&#8217;ve tweaked your Robots.txt file, test it to make sure it works right. You can use a tool like Google&#8217;s Search Console for this &#8212; <a href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\">https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool<\/a><\/p>\n<p><a href=\"http:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_3-1.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-full wp-image-754\" src=\"http:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_3-1.png\" alt=\"\" width=\"890\" height=\"431\" srcset=\"https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_3-1.png 890w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_3-1-300x145.png 300w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_3-1-768x372.png 768w\" sizes=\"(max-width: 890px) 100vw, 890px\" \/><\/a><\/p>\n<p>Or just open it in your browser to see if it looks okay &#8212; https:\/\/YourSite.com\/robots.txt<\/p>\n<h2>Common Mistakes<\/h2>\n<p>When editing your Robots.txt file, watch out for these common mistakes:<\/p>\n<p><strong>1. Misplacement of the Robots.txt File:<\/strong> The Robots.txt file should always be placed in the root directory. Placing it elsewhere will lead to it being ignored by bots.<\/p>\n<p><strong>2. Unnecessary Use of Trailing Slash:<\/strong> Adding a trailing slash (\/) when it&#8217;s not needed can lead to misinterpretation by bots. For example, &#171;Disallow: \/private\/&#187;might not be interpreted the same as &#171;Disallow: \/private&#187;.<\/p>\n<p><strong>3. Ignoring Case Sensitivity:<\/strong> URLs are case-sensitive, meaning &#171;\/My-Secret-Page&#187; is different from &#171;\/my-secret-page&#187;. This applies to directives in your Robots.txt file as well.<\/p>\n<p><strong>4. Blocking All Bots:<\/strong> A rule like &#171;User-agent: *&#187; followed by &#171;Disallow: \/&#187; blocks all bots from your entire website. Unless that&#8217;s your intention, avoid this setup.<\/p>\n<p><strong>5. Using Allow Directive Incorrectly:<\/strong> The &#171;Allow&#187; directive is not recognized by all bots, so using it as a complete access control measure can lead to problems.<\/p>\n<p><strong>6. Neglecting the Use of Comments:<\/strong> Forgetting to use &#171;#&#187; before comments can cause confusion for bots trying to read the file.<\/p>\n<p><strong>7. Using Noindex in Robots.txt:<\/strong> &#171;Noindex&#187; is a directive for individual web pages and is not recognized in a Robots.txt file.<\/p>\n<p><strong>8. Not Updating the Robots.txt File:<\/strong> Forgetting to update the Robots.txt file as your website evolves can lead to obsolete or incorrect rules.<\/p>\n<p><strong>9. Leaving a Testing Robots.txt Live:<\/strong> Always remember to replace a restrictive testing Robots.txt file with your regular one after you&#8217;re done testing.<\/p>\n<p class=\"\"><strong>10. Using Incorrect Syntax:<\/strong> Even small typos in directives like &#171;User-agent&#187; and &#171;Disallow&#187;, or in the paths you enter, can cause significant issues with how bots interpret your Robots.txt file.<\/p>\n<hr \/>\n<p>Remember, your Robots.txt is a powerful SEO tool. Take the time to set it up right and it can really help your PrestaShop site stand out in the world of eCommerce.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Robots.txt is a small, but powerful file that tells search engine bots how to interact with your website. It&#8217;s like a guidebook, showing them which pages to visit (index) and which ones to skip (ignore). A well-set-up Robots.txt can help your website&#8217;s SEO. Actually it&#8217;s not just for PrestaShop, it can be found on any &hellip; <a href=\"https:\/\/presta.site\/blog\/en\/a-simple-guide-to-configuring-robots-txt-in-prestashop\/\" class=\"more-link\">\u0427\u0438\u0442\u0430\u0442\u044c \u0434\u0430\u043b\u0435\u0435 <span class=\"screen-reader-text\">A Simple Guide to Configuring Robots.txt in PrestaShop<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts\/747"}],"collection":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/comments?post=747"}],"version-history":[{"count":6,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts\/747\/revisions"}],"predecessor-version":[{"id":756,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts\/747\/revisions\/756"}],"wp:attachment":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/media?parent=747"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/categories?post=747"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/tags?post=747"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}