[Last Call] Learn how to a build a cloud-first strategyRegister Now


htaccess robots (Google bot and others) question

Posted on 2007-07-28
Medium Priority
Last Modified: 2013-12-08
How do I block search engines to index content and files within a directory and all subdirectories.

Is that possible with only 1 .htaccess file??

The directories (and subdirectories) contains only images and PHP files...
Question by:mychel_normandeau
  • 2
LVL 33

Accepted Solution

humeniuk earned 750 total points
ID: 19585985
The proper way to do this is with a robots.txt file.  You can read more about them here - www.robotstxt.org/wc/robots.html - but in short, to block all robots from all directories, create a new document in Notepad with the following:

User-agent: *
Disallow: /

Then upload it to the root directory of your website.

There are other methods, but this is the most widely supported and most reliable.

Author Comment

ID: 19586084
I know, but theres spam bot that don't follow robots.txt...

when I make a .htaccess with their generator, can I place that .htaccess in a folder and all folders and sub-folders will be protected?

Content of the .htaccess
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot

<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
LVL 33

Assisted Solution

humeniuk earned 750 total points
ID: 19586387
Yes, those are different than the search engine robots.  The .htaccess approach you refer to will work - place it in the root folder and it will prevent access in subdirectories unless you place an .htaccess file in those specific directories that override it.

Featured Post

Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When crafting your “Why Us” page, there are a plethora of pitfalls to avoid. Follow these five tips, and you’ll be well on your way to creating an effective page.
Strategic internal linking is often considered an SEO power technique, especially for content marketing. Do you need to hire an SEO agency to optimize you internal linking? No, this article will help you understand the basics of internal linking and…
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
The is a quite short video tutorial. In this video, I'm going to show you how to create self-host WordPress blog with free hosting service.
Suggested Courses
Course of the Month18 days, 9 hours left to enroll

826 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question