Solved

Restrict from search engines?

Posted on 2007-12-05
8
340 Views
Last Modified: 2013-11-19
Is there a way, meta tags or otherwise, to keep the search engines from indexing a directory on my site?

Thank you in advance for your help and superlative brain power!

-- Scott
0
Comment
Question by:scottb50
8 Comments
 

Author Comment

by:scottb50
ID: 20416002
Just by answering, you officially rock.
0
 
LVL 33

Accepted Solution

by:
humeniuk earned 43 total points
ID: 20416413
The most effective way of doing this is with a robots.txt file - see www.robotstxt.org.

You can also use a noindex meta tag (www.robotstxt.org/meta.html), but is not quite as reliable as the robots.txt.
0
 
LVL 29

Assisted Solution

by:fibo
fibo earned 41 total points
ID: 20426480
Note that you have no real way to prevent all robots to spider this directory. Using robots.txt will give "clean robots" indication not to spider the directory... but it should be considered as a "red flag" for some inquisitive robots.
--> use exclusion by robots.txt, this will give the "well-behaving" robots (of which the lots important ones) indications of not indexing this directory and so not loosing useless time.
--> do NOT hope that this would protect a directory; so, if the directory content needs to be protected, use htaccess to restrict access; this would keep away "bad-behaving" (but well intentioned) robots from indexing your directory, and would keep away bad-intentioned spiders.
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 1

Assisted Solution

by:Server_Jockey
Server_Jockey earned 41 total points
ID: 20621453
Hi,

1. Create a file named robots.txt
2. Place the following inside the file

User-agent: *
Disallow: /

3. Save the file and upload it to the root of your website.

The two lines above basically denies any spiders or crawlers to your website.

Good Luck,

Server Jockey

*** url sig deleted per - www.experts-exchange.com/help.jsp#hi22 ***


0
 
LVL 29

Expert Comment

by:fibo
ID: 20698076
as an additional safety measure against curiosity, check that all of your "to be protected" directories do have an index.htm (or index.php, or...) that protects them.
The minimum  is an empty "index.htm" file which at least prevents http / html browsing of the directory and of its files.
Smarter is an index.htm file with a http AND javascript redirects to your home page.
Of course, if you already use an index.* file in this directory this cannot be done directly... but you might decide to rename this index.htm file (remember, it is opened by default if you go to the directory without specifying the page name) to some other difficult-to-guess name (eg, xedni.htm) AND putting a protective index.htm.

This would NOT protect you against direct access to a file in the directory (eg, a link coming from a "normal page" to a page in the directory) BUT at least it would prevent inquisitive badly-educated robots to explore all the file in a directory defined in a robots.txt file as a "secret directory".
0
 
LVL 29

Expert Comment

by:fibo
ID: 21008077
Scottb,

Remember closing the question by allocating the points to one or several of the experts whoo answered
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now