• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 677
  • Last Modified:

robots.txt subfolders - do not allow


I found this bit of code:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

I'm wondering if these lines will disallow the subfolders as well.

Thank you,
Victor Kimura
Victor Kimura
  • 2
3 Solutions
To learn more about robots.txt you should visit http://www.robotstxt.org/

However to answer your question, yes those lines will disallow access to subfolders also.
When using robots.txt please bear in mind that it is "world readable", i.e. may be accessed and viewed by anybody, not only robots.
Thus if you plan to keep search engines away from those folders on your domain that store information or data which should not be known by the public, then at the same time you will give hints to malicious intruders about the existence of such folders - which might provoke their curiosity.
Victor KimuraSEO, Web DeveloperAuthor Commented:
Hi torimar,

Thank you for the insight. Do you suggest another good way to prevent robots from accessing and indexing those folders? I'm asking more from a search engine optimization perspective as there are folders that would be beneficial if the robots do not access.

Much thanks,
Well, when it comes to SSL and canonicalization (http://www.mattcutts.com/blog/seo-advice-url-canonicalization/) then there is a nice rewrite trick explained here:

Other than that, I don't know any workaround. Maybe you want to read this on the topic: http://www.robotstxt.org/faq/nosecurity.html
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now