Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Restrict bots from crawling directories

Posted on 2010-08-16
6
Medium Priority
?
478 Views
Last Modified: 2012-05-10
Google has index one of my directories and showing in search results, to stop the directory and its sub directories I was thinking to restrict bots from accessing the directory, how can I do that?
0
Comment
Question by:sahanz
  • 3
  • 2
6 Comments
 
LVL 5

Accepted Solution

by:
stermeau earned 1000 total points
ID: 33444283
You can use a robots.txt file.
There are some simple examples here : http://www.robotstxt.org/orig.html

But you should also modify your web server configuration to disable directory listing.
0
 
LVL 7

Expert Comment

by:marektech
ID: 33444325
You could also us the following:

<meta name="robots" content="noindex,nofollow">

http://www.heritage-tech.net/188/alternative-to-using-robotstxt/
0
 
LVL 7

Expert Comment

by:marektech
ID: 33444340
More information about the robots metatag:

http://www.robotstxt.org/meta.html
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 1

Author Comment

by:sahanz
ID: 33445375
if I add those lines to the index file of the directory, will it stop from crawling sub directories?
0
 
LVL 7

Assisted Solution

by:marektech
marektech earned 1000 total points
ID: 33445538
You can use the robots.txt option on the root of your website and specify the directories which should be no go areas.

For example:

User-Agent: Googlebot
Disallow: /private/private.htm
Disallow: /secret/

Or via the Meta tag method the tag should be present on each page which is not to be indexed.

<meta name="robots" content="noindex,nofollow">
0
 
LVL 1

Author Closing Comment

by:sahanz
ID: 33455450
Thanks
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

FAQ pages provide a simple way for you to supply and for customers to find answers to the most common questions about your company. Here are six reasons why your company website should have a FAQ page
We take a look at the fast-evolving changes in Search Engine Optimization rules and algorithms by Google.
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.
Suggested Courses
Course of the Month10 days, 14 hours left to enroll

571 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question