A Simple Guide to Configuring PrestaShop robots.txt

A PrestaShop robots.txt file tells crawlers which paths they may fetch. It is crawl control, not a lock on indexing – and a bad rule can hide category pages or CSS while leaving thin URLs crawlable. This guide covers the Back Office generator, safe edits, and how to verify the live file without guessing.

From PrestaShop 1.7 onward (including 8 and 9), you regenerate the file under Shop Parameters → Traffic & SEO. Official screen notes are in the PrestaShop 9 Traffic documentation. Google’s own overview is in the robots.txt introduction – start there if you need the “what it cannot do” rules before you touch Disallow lines.

What PrestaShop robots.txt controls

robots.txt is a text file at the shop root (https://yoursite.com/robots.txt). Compliant bots read it before crawling. Use it to reduce crawl noise on admin paths, filters, or duplicate URL patterns – not to “hide” a page from Google Search.

If a URL must stay out of results, use noindex (meta or HTTP header) or access control. Blocking with robots.txt can still leave the URL listed without a snippet. That distinction is why many “I blocked it but it still shows” tickets are not really PrestaShop robots.txt bugs.

Default file after install

PrestaShop creates a robots.txt during install, tuned for a typical shop (admin folder, some module paths, and other crawl-heavy URLs). On a stock catalog with no custom staging folders, you often leave it alone.

Open /robots.txt in a private window after every regenerate. Confirm the file is reachable over HTTPS and that you did not accidentally Disallow / on a live shop.

When to edit (and when not to)

Edit when you have a concrete crawl problem:

• Staging or backup folders under the same host that bots should not fetch
• Custom paths that create near-duplicate crawl targets
• A missing or truncated file after a migration
• A Sitemap line that still points at an old domain

Skip edits when SEO is fine and you only copied a “block everything” snippet from a forum. Default PrestaShop robots.txt rules are usually enough until Search Console shows a real crawl issue.

Generate robots.txt from the Back Office

Prefer the BO generator over hand-editing the root file when you only need a clean default back:

1. Go to Shop Parameters → Traffic & SEO.
2. Scroll to Robots file generation.
3. Click Generate robots.txt file.

Regenerating overwrites the file on disk. If you added custom Disallow or Sitemap lines, copy them out first, regenerate, then re-apply carefully. On multistore, confirm you are in the right shop context before generating – see the PrestaShop multistore guide if shop switching is unclear.

Edit the file on disk

The live file sits in the shop root (same level as index.php). Use SFTP or the host file manager. Always keep a copy before saving – a single typo in Disallow can block theme assets or the whole tree.

If friendly URLs break after other SEO changes, fix rewriting first with the PrestaShop URL rewriting checklist; robots.txt will not repair 404s from a missing .htaccess.

Basic syntax merchants actually use

Each block starts with a User-agent line, then one or more Disallow / Allow rules:

Block one directory for all bots:

User-agent: *
Disallow: /private

Block everything (staging only – never leave this on production):

User-agent: *
Disallow: /

Point crawlers at your XML sitemap:

Sitemap: https://example.com/sitemap.xml

After you add a Sitemap line, make sure the XML itself is generated and submitted – walk through that in the PrestaShop sitemap guide.

Test PrestaShop robots.txt after every change

Do not rely on the old standalone “robots.txt tester” URL – Google retired that tool. Use this instead:

• Open https://yoursite.com/robots.txt and confirm the text you saved is what the public sees.
• In Google Search Console, use URL Inspection on a key product or category URL and check crawl allow/block signals after the next fetch.
• Watch Page Indexing / crawl reports for sudden spikes of “Blocked by robots.txt” on URLs you still want crawled.

Common PrestaShop robots.txt mistakes

• File not in the root – bots only honor /robots.txt on that host. A file under /shop/robots.txt does not protect the apex domain.
• Disallow: / left on live – common after staging clones. Regenerate or restore the production file immediately.
• Assuming robots.txt = secret pages – anything truly private needs login or noindex, not only Disallow.
• Noindex lines in robots.txt – Google does not treat that as a supported way to deindex HTML pages.
• Wrong Sitemap URL after domain change – HTTPS and hostname must match the live shop.
• Case and trailing-slash mismatches – paths are case-sensitive; /Private is not /private.
• Hand edits wiped by Generate – regenerate last, or re-apply customs afterward.

Quick checklist

1. View the live /robots.txt in a browser.
2. If the file is missing or corrupted, generate it from Traffic & SEO.
3. Only add Disallow rules you can name a reason for.
4. Keep or fix the Sitemap line; verify XML separately.
5. Re-check a money page in Search Console after changes.
6. Never ship a full-site Disallow from staging to production.

Treat PrestaShop robots.txt as a crawl budget and noise filter. Regenerate from the BO when you need a clean baseline, add Disallow lines only for named crawl problems, and re-check /robots.txt in a private window before you close the ticket.