Solved

Restrict from search engines?

Posted on 2007-12-05
8
349 Views
Last Modified: 2013-11-19
Is there a way, meta tags or otherwise, to keep the search engines from indexing a directory on my site?

Thank you in advance for your help and superlative brain power!

-- Scott
0
Comment
Question by:scottb50
8 Comments
 

Author Comment

by:scottb50
ID: 20416002
Just by answering, you officially rock.
0
 
LVL 33

Accepted Solution

by:
humeniuk earned 43 total points
ID: 20416413
The most effective way of doing this is with a robots.txt file - see www.robotstxt.org.

You can also use a noindex meta tag (www.robotstxt.org/meta.html), but is not quite as reliable as the robots.txt.
0
 
LVL 29

Assisted Solution

by:fibo
fibo earned 41 total points
ID: 20426480
Note that you have no real way to prevent all robots to spider this directory. Using robots.txt will give "clean robots" indication not to spider the directory... but it should be considered as a "red flag" for some inquisitive robots.
--> use exclusion by robots.txt, this will give the "well-behaving" robots (of which the lots important ones) indications of not indexing this directory and so not loosing useless time.
--> do NOT hope that this would protect a directory; so, if the directory content needs to be protected, use htaccess to restrict access; this would keep away "bad-behaving" (but well intentioned) robots from indexing your directory, and would keep away bad-intentioned spiders.
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 
LVL 1

Assisted Solution

by:Server_Jockey
Server_Jockey earned 41 total points
ID: 20621453
Hi,

1. Create a file named robots.txt
2. Place the following inside the file

User-agent: *
Disallow: /

3. Save the file and upload it to the root of your website.

The two lines above basically denies any spiders or crawlers to your website.

Good Luck,

Server Jockey

*** url sig deleted per - www.experts-exchange.com/help.jsp#hi22 ***


0
 
LVL 29

Expert Comment

by:fibo
ID: 20698076
as an additional safety measure against curiosity, check that all of your "to be protected" directories do have an index.htm (or index.php, or...) that protects them.
The minimum  is an empty "index.htm" file which at least prevents http / html browsing of the directory and of its files.
Smarter is an index.htm file with a http AND javascript redirects to your home page.
Of course, if you already use an index.* file in this directory this cannot be done directly... but you might decide to rename this index.htm file (remember, it is opened by default if you go to the directory without specifying the page name) to some other difficult-to-guess name (eg, xedni.htm) AND putting a protective index.htm.

This would NOT protect you against direct access to a file in the directory (eg, a link coming from a "normal page" to a page in the directory) BUT at least it would prevent inquisitive badly-educated robots to explore all the file in a directory defined in a robots.txt file as a "secret directory".
0
 
LVL 29

Expert Comment

by:fibo
ID: 21008077
Scottb,

Remember closing the question by allocating the points to one or several of the experts whoo answered
0

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Marketing can be an uncomfortable undertaking, especially if your material is technology based. Luckily, we’ve compiled some simple and (relatively) painless tips to put an end to your trepidation and start your path to success.
This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…

825 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question