• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1147
  • Last Modified:

htaccess robots (Google bot and others) question

How do I block search engines to index content and files within a directory and all subdirectories.

Is that possible with only 1 .htaccess file??

The directories (and subdirectories) contains only images and PHP files...
0
mychel_normandeau
Asked:
mychel_normandeau
  • 2
2 Solutions
 
humeniukCommented:
The proper way to do this is with a robots.txt file.  You can read more about them here - www.robotstxt.org/wc/robots.html - but in short, to block all robots from all directories, create a new document in Notepad with the following:

User-agent: *
Disallow: /

Then upload it to the root directory of your website.

There are other methods, but this is the most widely supported and most reliable.
0
 
mychel_normandeauAuthor Commented:
I know, but theres spam bot that don't follow robots.txt...

See
http://www.clockwatchers.com/robots_bad.html
when I make a .htaccess with their generator, can I place that .htaccess in a folder and all folders and sub-folders will be protected?

Content of the .htaccess
---
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot

<Limit GET POST>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
---
0
 
humeniukCommented:
Yes, those are different than the search engine robots.  The .htaccess approach you refer to will work - place it in the root folder and it will prevent access in subdirectories unless you place an .htaccess file in those specific directories that override it.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now