Why You Should Never Use a Generic robots.txt File (with Platform-Specific Templates)

Jinnat Ul Hasan

By Jinnat Ul Hasan

In Technical SEO

On 2nd July 2025

If you care about ranking in Google, Google AI Overviews, ChatGPT, or any AI-powered search, you can’t afford to rely on a generic robots.txt file. I’ve worked on sites across WordPress, Shopify, Adobe Experience Manager (AEM), Magento, Wix, and more—and I’ve seen firsthand how generic or default robots.txt setups create SEO headaches, block important content, or leak sensitive areas.

Let me show you why I always use a custom, up-to-date robots.txt file—and how you can do the same. Remember, this is just the templates; for additional settings, you should always consult with technical seo professionals.

Why I Don’t Trust Generic robots.txt Files

  • Every platform is different: What’s dangerous to index in WordPress might be essential in Shopify or AEM.
  • Blocking the wrong content: I’ve seen default robots.txt rules block vital CSS and JS files, which can break how your site appears in Google or AI models.
  • Duplicate content risk: Generic files often lack clean URL parameters, archive pages, or search results.
  • Security: Too often, admin, config, or authoring areas are exposed to Google and public search if you don’t disallow them specifically.
  • Missed sitemap links: I always add a sitemap directive; generic files usually forget.

How I Build robots.txt Files (Best Practices)

  • Relative paths only, no hardcoded URLs.
  • Always allow CSS, JS, and images unless there’s a good reason to block.
  • Reference your sitemap at the end.
  • Don’t rely on Disallow for de-indexing, use noindex meta tags for that.
  • Test in Google Search Console (or a similar tool) every time.
  • Add comments so anyone can understand why a rule exists.

My robots.txt Templates for Top Platforms

Feel free to adapt these to your own site—they’re the templates I use as a starting point.

WordPress

I block admin, system, and archive pages but keep AJAX open for front-end features. I always link my sitemap. Additionally, you might opt out of blocking authors (/author/) and tags (/tags/) from being crawled and indexed.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-json/
Disallow: /*?
Disallow: /trackback/
Disallow: /feed/
Disallow: /search/
Disallow: /xmlrpc.php
Disallow: /readme.html
Disallow: /license.txt

Sitemap: https://www.yourdomain.com/sitemap_index.xml

WordPress with WooCommerce

WooCommerce shops generate a large number of duplicate and transactional URLs. I block those to keep Google focused on real products and categories.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=*
Disallow: /*orderby=*
Disallow: /*min_price=*
Disallow: /*max_price=*
Disallow: /*filter_*

Sitemap: https://www.yourdomain.com/sitemap_index.xml

Shopify

Shopify robots.txt file settings are notorious for complexity and issues. However, you should start your custom robots.txt file using this template.

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /orders
Disallow: /search
Disallow: /collections/*+*
Disallow: /collections/*filter*

Sitemap: https://www.yourdomain.com/sitemap.xml

Wix

Wix writes most of this for you. I add custom rules only if the platform allows.

User-agent: *
Disallow: /_api/
Disallow: /*.pdf$
Disallow: /my-private-page/

Sitemap: https://www.yourdomain.com/sitemap.xml

Magento / Custom E-commerce

I keep bots out of customer, cart, and internal search, and I eliminate duplicate URLs on every e-commerce site I manage.

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /cart/
Disallow: /customer/
Disallow: /catalogsearch/
Disallow: /review/
Disallow: /*?q=*
Disallow: /*?cat=*
Disallow: /*_sid=

Sitemap: https://www.yourdomain.com/sitemap.xml

Adobe Experience Manager (AEM)

AEM has lots of internal folders and authoring tools. I always block those and ensure Google can fetch real CSS, JS, and images for optimal rendering.

User-agent: *
Disallow: /libs/
Disallow: /etc/
Disallow: /system/
Disallow: /bin/
Disallow: /crx/
Disallow: /crx/de/
Disallow: /siteadmin/
Disallow: /useradmin/
Disallow: /welcome
Disallow: /index.jsp
Disallow: /editor.html/
Disallow: /preview/
Disallow: /stage/
Disallow: /test/
Disallow: /content/dam/.*/renditions/

Allow: /content/dam/*.css$
Allow: /content/dam/*.js$
Allow: /content/dam/*.jpg$
Allow: /content/dam/*.png$
Allow: /content/dam/*.svg$

Sitemap: https://www.yourdomain.com/sitemap.xml

robots.txt Questions I Get Asked All the Time

Should I block CSS or JavaScript?

No! Search engines and AI search need these to render and understand your pages.

Does Disallow mean my page disappears from Google?

Not by itself. It only stops bots from crawling further. If it’s already in Google, you need a noindex tag.

Why bother with a sitemap link?

It’s the easiest way for Google, Bing, and AI-powered search to find all your important URLs, especially as AI search gets smarter.

Final Words

If you’re using the default robots.txt file as everyone else, you’re likely hurting your website’s SEO performance, making your site less secure, and making it clunky. I update robots.txt for every client, every time, because it matters for better crawling, indexing, and future-proofing the website.