[Webinar] Streamline your web hosting managementRegister Today

x
?
Solved

Restrict from search engines?

Posted on 2007-12-05
8
Medium Priority
?
368 Views
Last Modified: 2013-11-19
Is there a way, meta tags or otherwise, to keep the search engines from indexing a directory on my site?

Thank you in advance for your help and superlative brain power!

-- Scott
0
Comment
Question by:scottb50
6 Comments
 

Author Comment

by:scottb50
ID: 20416002
Just by answering, you officially rock.
0
 
LVL 33

Accepted Solution

by:
humeniuk earned 172 total points
ID: 20416413
The most effective way of doing this is with a robots.txt file - see www.robotstxt.org.

You can also use a noindex meta tag (www.robotstxt.org/meta.html), but is not quite as reliable as the robots.txt.
0
 
LVL 29

Assisted Solution

by:fibo
fibo earned 164 total points
ID: 20426480
Note that you have no real way to prevent all robots to spider this directory. Using robots.txt will give "clean robots" indication not to spider the directory... but it should be considered as a "red flag" for some inquisitive robots.
--> use exclusion by robots.txt, this will give the "well-behaving" robots (of which the lots important ones) indications of not indexing this directory and so not loosing useless time.
--> do NOT hope that this would protect a directory; so, if the directory content needs to be protected, use htaccess to restrict access; this would keep away "bad-behaving" (but well intentioned) robots from indexing your directory, and would keep away bad-intentioned spiders.
0
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

 
LVL 1

Assisted Solution

by:Server_Jockey
Server_Jockey earned 164 total points
ID: 20621453
Hi,

1. Create a file named robots.txt
2. Place the following inside the file

User-agent: *
Disallow: /

3. Save the file and upload it to the root of your website.

The two lines above basically denies any spiders or crawlers to your website.

Good Luck,

Server Jockey

*** url sig deleted per - www.experts-exchange.com/help.jsp#hi22 ***


0
 
LVL 29

Expert Comment

by:fibo
ID: 20698076
as an additional safety measure against curiosity, check that all of your "to be protected" directories do have an index.htm (or index.php, or...) that protects them.
The minimum  is an empty "index.htm" file which at least prevents http / html browsing of the directory and of its files.
Smarter is an index.htm file with a http AND javascript redirects to your home page.
Of course, if you already use an index.* file in this directory this cannot be done directly... but you might decide to rename this index.htm file (remember, it is opened by default if you go to the directory without specifying the page name) to some other difficult-to-guess name (eg, xedni.htm) AND putting a protective index.htm.

This would NOT protect you against direct access to a file in the directory (eg, a link coming from a "normal page" to a page in the directory) BUT at least it would prevent inquisitive badly-educated robots to explore all the file in a directory defined in a robots.txt file as a "secret directory".
0
 
LVL 29

Expert Comment

by:fibo
ID: 21008077
Scottb,

Remember closing the question by allocating the points to one or several of the experts whoo answered
0

Featured Post

The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

No other job is as rewarding and demanding as building an iPhone app is. It is not really in the hands of the developer for the success of an iPhone app. Many factors operate jointly for every iOS application's success in the market.
How does storytelling impact the web? What goes into a great story on the web? And how can it be used for marketing purposes? All of these questions will be answered.
An overview of how to create reports in Adobe Analytics (formerly Omniture Site Catalyst) using pageNames, events, eVars and props. This video will show you how to install the Omniture Debugger tool so can see (and test) what is being passed int…
Learn how to set-up PayPal payment integration in your Wufoo form. Allow your users to remit payment through PayPal upon completion of your online form. This is helpful for collecting membership payments, customer payments, donations, and more.
Suggested Courses

608 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question