Solved

Restrict bots from crawling directories

Posted on 2010-08-16
6
468 Views
Last Modified: 2012-05-10
Google has index one of my directories and showing in search results, to stop the directory and its sub directories I was thinking to restrict bots from accessing the directory, how can I do that?
0
Comment
Question by:sahanz
  • 3
  • 2
6 Comments
 
LVL 5

Accepted Solution

by:
stermeau earned 250 total points
ID: 33444283
You can use a robots.txt file.
There are some simple examples here : http://www.robotstxt.org/orig.html

But you should also modify your web server configuration to disable directory listing.
0
 
LVL 7

Expert Comment

by:marektech
ID: 33444325
You could also us the following:

<meta name="robots" content="noindex,nofollow">

http://www.heritage-tech.net/188/alternative-to-using-robotstxt/
0
 
LVL 7

Expert Comment

by:marektech
ID: 33444340
More information about the robots metatag:

http://www.robotstxt.org/meta.html
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 
LVL 1

Author Comment

by:sahanz
ID: 33445375
if I add those lines to the index file of the directory, will it stop from crawling sub directories?
0
 
LVL 7

Assisted Solution

by:marektech
marektech earned 250 total points
ID: 33445538
You can use the robots.txt option on the root of your website and specify the directories which should be no go areas.

For example:

User-Agent: Googlebot
Disallow: /private/private.htm
Disallow: /secret/

Or via the Meta tag method the tag should be present on each page which is not to be indexed.

<meta name="robots" content="noindex,nofollow">
0
 
LVL 1

Author Closing Comment

by:sahanz
ID: 33455450
Thanks
0

Featured Post

VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you are a web developer, you would be aware of the <iframe> tag in HTML. The <iframe> stands for inline frame and is used to embed another document within the current HTML document. The embedded document could be even another website.
A/B testing is a simple and effective trick to get to know your audience, increase website conversions and make the most out of your online ad campaigns. It's widely available and doesn't need much tech knowledge to be executed, but the results it y…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.

808 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question