Solved

Restrict from search engines?

Posted on 2007-12-05
8
352 Views
Last Modified: 2013-11-19
Is there a way, meta tags or otherwise, to keep the search engines from indexing a directory on my site?

Thank you in advance for your help and superlative brain power!

-- Scott
0
Comment
Question by:scottb50
8 Comments
 

Author Comment

by:scottb50
ID: 20416002
Just by answering, you officially rock.
0
 
LVL 33

Accepted Solution

by:
humeniuk earned 43 total points
ID: 20416413
The most effective way of doing this is with a robots.txt file - see www.robotstxt.org.

You can also use a noindex meta tag (www.robotstxt.org/meta.html), but is not quite as reliable as the robots.txt.
0
 
LVL 29

Assisted Solution

by:fibo
fibo earned 41 total points
ID: 20426480
Note that you have no real way to prevent all robots to spider this directory. Using robots.txt will give "clean robots" indication not to spider the directory... but it should be considered as a "red flag" for some inquisitive robots.
--> use exclusion by robots.txt, this will give the "well-behaving" robots (of which the lots important ones) indications of not indexing this directory and so not loosing useless time.
--> do NOT hope that this would protect a directory; so, if the directory content needs to be protected, use htaccess to restrict access; this would keep away "bad-behaving" (but well intentioned) robots from indexing your directory, and would keep away bad-intentioned spiders.
0
MIM Survival Guide for Service Desk Managers

Major incidents can send mastered service desk processes into disorder. Systems and tools produce the data needed to resolve these incidents, but your challenge is getting that information to the right people fast. Check out the Survival Guide and begin bringing order to chaos.

 
LVL 1

Assisted Solution

by:Server_Jockey
Server_Jockey earned 41 total points
ID: 20621453
Hi,

1. Create a file named robots.txt
2. Place the following inside the file

User-agent: *
Disallow: /

3. Save the file and upload it to the root of your website.

The two lines above basically denies any spiders or crawlers to your website.

Good Luck,

Server Jockey

*** url sig deleted per - www.experts-exchange.com/help.jsp#hi22 ***


0
 
LVL 29

Expert Comment

by:fibo
ID: 20698076
as an additional safety measure against curiosity, check that all of your "to be protected" directories do have an index.htm (or index.php, or...) that protects them.
The minimum  is an empty "index.htm" file which at least prevents http / html browsing of the directory and of its files.
Smarter is an index.htm file with a http AND javascript redirects to your home page.
Of course, if you already use an index.* file in this directory this cannot be done directly... but you might decide to rename this index.htm file (remember, it is opened by default if you go to the directory without specifying the page name) to some other difficult-to-guess name (eg, xedni.htm) AND putting a protective index.htm.

This would NOT protect you against direct access to a file in the directory (eg, a link coming from a "normal page" to a page in the directory) BUT at least it would prevent inquisitive badly-educated robots to explore all the file in a directory defined in a robots.txt file as a "secret directory".
0
 
LVL 29

Expert Comment

by:fibo
ID: 21008077
Scottb,

Remember closing the question by allocating the points to one or several of the experts whoo answered
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Today, the web development industry is booming, and many people consider it to be their vocation. The question you may be asking yourself is – how do I become a web developer?
Starting your own business is always a daunting process, and for most people it is brand new experience. Avoid the common pitfalls by following these tips to start on the road to success.
The viewer will learn how to count occurrences of each item in an array.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question