• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 673
  • Last Modified:

robots.txt subfolders - do not allow


I found this bit of code:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

I'm wondering if these lines will disallow the subfolders as well.

Thank you,
Victor Kimura
Victor Kimura
  • 2
3 Solutions
To learn more about robots.txt you should visit http://www.robotstxt.org/

However to answer your question, yes those lines will disallow access to subfolders also.
When using robots.txt please bear in mind that it is "world readable", i.e. may be accessed and viewed by anybody, not only robots.
Thus if you plan to keep search engines away from those folders on your domain that store information or data which should not be known by the public, then at the same time you will give hints to malicious intruders about the existence of such folders - which might provoke their curiosity.
Victor KimuraAuthor Commented:
Hi torimar,

Thank you for the insight. Do you suggest another good way to prevent robots from accessing and indexing those folders? I'm asking more from a search engine optimization perspective as there are folders that would be beneficial if the robots do not access.

Much thanks,
Well, when it comes to SSL and canonicalization (http://www.mattcutts.com/blog/seo-advice-url-canonicalization/) then there is a nice rewrite trick explained here:

Other than that, I don't know any workaround. Maybe you want to read this on the topic: http://www.robotstxt.org/faq/nosecurity.html

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now