robots.txt subfolders - do not allow

Posted on 2009-12-23
Last Modified: 2013-12-15

I found this bit of code:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

I'm wondering if these lines will disallow the subfolders as well.

Thank you,
Question by:Victor Kimura
    LVL 57

    Assisted Solution

    To learn more about robots.txt you should visit

    However to answer your question, yes those lines will disallow access to subfolders also.
    LVL 35

    Accepted Solution

    When using robots.txt please bear in mind that it is "world readable", i.e. may be accessed and viewed by anybody, not only robots.
    Thus if you plan to keep search engines away from those folders on your domain that store information or data which should not be known by the public, then at the same time you will give hints to malicious intruders about the existence of such folders - which might provoke their curiosity.

    Author Comment

    by:Victor Kimura
    Hi torimar,

    Thank you for the insight. Do you suggest another good way to prevent robots from accessing and indexing those folders? I'm asking more from a search engine optimization perspective as there are folders that would be beneficial if the robots do not access.

    Much thanks,
    LVL 35

    Assisted Solution

    Well, when it comes to SSL and canonicalization ( then there is a nice rewrite trick explained here:

    Other than that, I don't know any workaround. Maybe you want to read this on the topic:

    Featured Post

    Do You Know the 4 Main Threat Actor Types?

    Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

    Join & Write a Comment

    If your site has a few sections that need to be secure when data is transmitted between the server and local computer, such as a /order/ section for ordering or /customer/ which contains customer data, etc it would of course be recommended to secure…
    If your business is like most, chances are you still need to maintain a fax infrastructure for your staff. It’s hard to believe that a communication technology that was thriving in the mid-80s could still be an essential part of your team’s modern I…
    After creating this article (, I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
    Here's a very brief overview of the methods PRTG Network Monitor ( offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

    734 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    25 Experts available now in Live!

    Get 1:1 Help Now