{"id":747,"date":"2023-06-04T11:47:26","date_gmt":"2023-06-04T08:47:26","guid":{"rendered":"https:\/\/presta.site\/blog\/?p=747"},"modified":"2026-07-26T08:41:20","modified_gmt":"2026-07-26T05:41:20","slug":"a-simple-guide-to-configuring-robots-txt-in-prestashop","status":"publish","type":"post","link":"https:\/\/presta.site\/blog\/en\/a-simple-guide-to-configuring-robots-txt-in-prestashop\/","title":{"rendered":"A Simple Guide to Configuring PrestaShop robots.txt"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A <strong>PrestaShop robots.txt<\/strong> file tells crawlers which paths they may fetch. It is crawl control, not a lock on indexing &#8211; and a bad rule can hide category pages or CSS while leaving thin URLs crawlable. This guide covers the Back Office generator, safe edits, and how to verify the live file without guessing.<\/p>\n\n\n<!--more-->\n\n\n<p class=\"wp-block-paragraph\">From PrestaShop 1.7 onward (including 8 and 9), you regenerate the file under <strong>Shop Parameters \u2192 Traffic &amp; SEO<\/strong>. Official screen notes are in the <a href=\"https:\/\/docs.prestashop-project.org\/v.9-documentation\/user-guide\/configuring-shop\/shop-parameters\/traffic\" target=\"_blank\" rel=\"noopener noreferrer\">PrestaShop 9 Traffic documentation<\/a>. Google\u2019s own overview is in the <a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener noreferrer\">robots.txt introduction<\/a> &#8211; start there if you need the \u201cwhat it cannot do\u201d rules before you touch Disallow lines.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What PrestaShop robots.txt controls<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">robots.txt is a text file at the shop root (<code class=\"codecolorer text default\"><span class=\"text\">https:\/\/yoursite.com\/robots.txt<\/span><\/code>). Compliant bots read it before crawling. Use it to reduce crawl noise on admin paths, filters, or duplicate URL patterns &#8211; not to \u201chide\u201d a page from Google Search.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If a URL must stay out of results, use <code class=\"codecolorer text default\"><span class=\"text\">noindex<\/span><\/code> (meta or HTTP header) or access control. Blocking with robots.txt can still leave the URL listed without a snippet. That distinction is why many \u201cI blocked it but it still shows\u201d tickets are not really PrestaShop robots.txt bugs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Default file after install<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">PrestaShop creates a robots.txt during install, tuned for a typical shop (admin folder, some module paths, and other crawl-heavy URLs). On a stock catalog with no custom staging folders, you often leave it alone.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"737\" height=\"436\" src=\"https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_11.png\" alt=\"Example PrestaShop robots.txt file in the browser\" class=\"wp-image-752\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Open <code class=\"codecolorer text default\"><span class=\"text\">\/robots.txt<\/span><\/code> in a private window after every regenerate. Confirm the file is reachable over HTTPS and that you did not accidentally Disallow <code class=\"codecolorer text default\"><span class=\"text\">\/<\/span><\/code> on a live shop.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">When to edit (and when not to)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Edit when you have a concrete crawl problem:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Staging or backup folders under the same host that bots should not fetch<\/li>\n<li>Custom paths that create near-duplicate crawl targets<\/li>\n<li>A missing or truncated file after a migration<\/li>\n<li>A Sitemap line that still points at an old domain<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Skip edits when SEO is fine and you only copied a \u201cblock everything\u201d snippet from a forum. Default PrestaShop robots.txt rules are usually enough until Search Console shows a real crawl issue.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Generate robots.txt from the Back Office<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Prefer the BO generator over hand-editing the root file when you only need a clean default back:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Go to <strong>Shop Parameters \u2192 Traffic &amp; SEO<\/strong>.<\/li>\n<li>Scroll to <strong>Robots file generation<\/strong>.<\/li>\n<li>Click <strong>Generate robots.txt file<\/strong>.<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"205\" src=\"https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-1024x205.png\" alt=\"PrestaShop Traffic and SEO robots.txt generation button\" class=\"wp-image-753\" srcset=\"https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-1024x205.png 1024w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-300x60.png 300w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1-768x154.png 768w, https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_2-1.png 1354w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Regenerating overwrites the file on disk. If you added custom Disallow or Sitemap lines, copy them out first, regenerate, then re-apply carefully. On multistore, confirm you are in the right shop context before generating &#8211; see the <a href=\"https:\/\/presta.site\/blog\/en\/prestashop-multistore\/\">PrestaShop multistore guide<\/a> if shop switching is unclear.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Edit the file on disk<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The live file sits in the <strong>shop root<\/strong> (same level as <code class=\"codecolorer text default\"><span class=\"text\">index.php<\/span><\/code>). Use SFTP or the host file manager. Always keep a copy before saving &#8211; a single typo in <code class=\"codecolorer text default\"><span class=\"text\">Disallow<\/span><\/code> can block theme assets or the whole tree.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If friendly URLs break after other SEO changes, fix rewriting first with the <a href=\"https:\/\/presta.site\/blog\/en\/solving-prestashop-seo-url-rewriting-problems\/\">PrestaShop URL rewriting checklist<\/a>; robots.txt will not repair 404s from a missing <code class=\"codecolorer text default\"><span class=\"text\">.htaccess<\/span><\/code>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Basic syntax merchants actually use<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Each block starts with a <code class=\"codecolorer text default\"><span class=\"text\">User-agent<\/span><\/code> line, then one or more <code class=\"codecolorer text default\"><span class=\"text\">Disallow<\/span><\/code> \/ <code class=\"codecolorer text default\"><span class=\"text\">Allow<\/span><\/code> rules:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Block one directory for all bots:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: *\nDisallow: \/private<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Block everything (staging only &#8211; never leave this on production):<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: *\nDisallow: \/<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Point crawlers at your XML sitemap:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Sitemap: https:\/\/example.com\/sitemap.xml<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After you add a Sitemap line, make sure the XML itself is generated and submitted &#8211; walk through that in the <a href=\"https:\/\/presta.site\/blog\/en\/what-is-sitemap-and-how-to-work-with-it-in-prestashop\/\">PrestaShop sitemap guide<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Test PrestaShop robots.txt after every change<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Do not rely on the old standalone \u201crobots.txt tester\u201d URL &#8211; Google retired that tool. Use this instead:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Open <code class=\"codecolorer text default\"><span class=\"text\">https:\/\/yoursite.com\/robots.txt<\/span><\/code> and confirm the text you saved is what the public sees.<\/li>\n<li>In Google Search Console, use <strong>URL Inspection<\/strong> on a key product or category URL and check crawl allow\/block signals after the next fetch.<\/li>\n<li>Watch Page Indexing \/ crawl reports for sudden spikes of \u201cBlocked by robots.txt\u201d on URLs you still want crawled.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"890\" height=\"431\" src=\"https:\/\/presta.site\/blog\/wp-content\/uploads\/2023\/06\/Screenshot_3-1.png\" alt=\"Checking PrestaShop robots.txt rules after a change\" class=\"wp-image-754\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Common PrestaShop robots.txt mistakes<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>File not in the root<\/strong> &#8211; bots only honor <code class=\"codecolorer text default\"><span class=\"text\">\/robots.txt<\/span><\/code> on that host. A file under <code class=\"codecolorer text default\"><span class=\"text\">\/shop\/robots.txt<\/span><\/code> does not protect the apex domain.<\/li>\n<li><strong><code class=\"codecolorer text default\"><span class=\"text\">Disallow: \/<\/span><\/code> left on live<\/strong> &#8211; common after staging clones. Regenerate or restore the production file immediately.<\/li>\n<li><strong>Assuming robots.txt = secret pages<\/strong> &#8211; anything truly private needs login or <code class=\"codecolorer text default\"><span class=\"text\">noindex<\/span><\/code>, not only Disallow.<\/li>\n<li><strong><code class=\"codecolorer text default\"><span class=\"text\">Noindex<\/span><\/code> lines in robots.txt<\/strong> &#8211; Google does not treat that as a supported way to deindex HTML pages.<\/li>\n<li><strong>Wrong Sitemap URL after domain change<\/strong> &#8211; HTTPS and hostname must match the live shop.<\/li>\n<li><strong>Case and trailing-slash mismatches<\/strong> &#8211; paths are case-sensitive; <code class=\"codecolorer text default\"><span class=\"text\">\/Private<\/span><\/code> is not <code class=\"codecolorer text default\"><span class=\"text\">\/private<\/span><\/code>.<\/li>\n<li><strong>Hand edits wiped by Generate<\/strong> &#8211; regenerate last, or re-apply customs afterward.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Quick checklist<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>View the live <code class=\"codecolorer text default\"><span class=\"text\">\/robots.txt<\/span><\/code> in a browser.<\/li>\n<li>If the file is missing or corrupted, generate it from Traffic &amp; SEO.<\/li>\n<li>Only add Disallow rules you can name a reason for.<\/li>\n<li>Keep or fix the Sitemap line; verify XML separately.<\/li>\n<li>Re-check a money page in Search Console after changes.<\/li>\n<li>Never ship a full-site Disallow from staging to production.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Treat PrestaShop robots.txt as a crawl budget and noise filter. Regenerate from the BO when you need a clean baseline, add Disallow lines only for named crawl problems, and re-check <code class=\"codecolorer text default\"><span class=\"text\">\/robots.txt<\/span><\/code> in a private window before you close the ticket.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>A simple guide to configuring PrestaShop robots.txt from the Back Office generator, with safe Disallow edits and live-file checks so you do not block crawl by mistake.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[],"class_list":["post-747","post","type-post","status-publish","format-standard","hentry","category-development"],"_links":{"self":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts\/747","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/comments?post=747"}],"version-history":[{"count":8,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts\/747\/revisions"}],"predecessor-version":[{"id":2015,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/posts\/747\/revisions\/2015"}],"wp:attachment":[{"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/media?parent=747"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/categories?post=747"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/presta.site\/blog\/wp-json\/wp\/v2\/tags?post=747"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}