htaccess robots (Google bot and others) question

Posted on 2007-07-28
Last Modified: 2013-12-08
How do I block search engines to index content and files within a directory and all subdirectories.

Is that possible with only 1 .htaccess file??

The directories (and subdirectories) contains only images and PHP files...
Question by:mychel_normandeau
    LVL 33

    Accepted Solution

    The proper way to do this is with a robots.txt file.  You can read more about them here - - but in short, to block all robots from all directories, create a new document in Notepad with the following:

    User-agent: *
    Disallow: /

    Then upload it to the root directory of your website.

    There are other methods, but this is the most widely supported and most reliable.

    Author Comment

    I know, but theres spam bot that don't follow robots.txt...

    when I make a .htaccess with their generator, can I place that .htaccess in a folder and all folders and sub-folders will be protected?

    Content of the .htaccess
    SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
    SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
    SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
    SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
    SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
    SetEnvIfNoCase User-Agent "^Teleport" bad_bot
    SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
    SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
    SetEnvIfNoCase User-Agent "^Zeus" bad_bot

    <Limit GET POST>
    Order Allow,Deny
    Allow from all
    Deny from env=bad_bot
    LVL 33

    Assisted Solution

    Yes, those are different than the search engine robots.  The .htaccess approach you refer to will work - place it in the root folder and it will prevent access in subdirectories unless you place an .htaccess file in those specific directories that override it.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    IT, Stop Being Called Into Every Meeting

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    Suggested Solutions

    Title # Comments Views Activity
    Problem with .htaccess file 4 47
    PHP Register global 21 59
    Dreamweaver server behavior gone 6 50
    Effect on button 11 39
    Uploading files to the web server has become common part of almost any kind of web application. People use different technologies to solve this, but regardless of the technology used, it is always useful to have some kind of progress indicator shown…
    Foreword (July, 2015) Since I first wrote this article, years ago, a great many more people have begun using the internet.  They are coming online from every part of the globe, learning, reading, shopping and spending money at an ever-increasing ra…
    This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
    Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now