Solved

Restrict bots from crawling directories

Posted on 2010-08-16
6
465 Views
Last Modified: 2012-05-10
Google has index one of my directories and showing in search results, to stop the directory and its sub directories I was thinking to restrict bots from accessing the directory, how can I do that?
0
Comment
Question by:sahanz
  • 3
  • 2
6 Comments
 
LVL 5

Accepted Solution

by:
stermeau earned 250 total points
ID: 33444283
You can use a robots.txt file.
There are some simple examples here : http://www.robotstxt.org/orig.html

But you should also modify your web server configuration to disable directory listing.
0
 
LVL 7

Expert Comment

by:marektech
ID: 33444325
You could also us the following:

<meta name="robots" content="noindex,nofollow">

http://www.heritage-tech.net/188/alternative-to-using-robotstxt/
0
 
LVL 7

Expert Comment

by:marektech
ID: 33444340
More information about the robots metatag:

http://www.robotstxt.org/meta.html
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 1

Author Comment

by:sahanz
ID: 33445375
if I add those lines to the index file of the directory, will it stop from crawling sub directories?
0
 
LVL 7

Assisted Solution

by:marektech
marektech earned 250 total points
ID: 33445538
You can use the robots.txt option on the root of your website and specify the directories which should be no go areas.

For example:

User-Agent: Googlebot
Disallow: /private/private.htm
Disallow: /secret/

Or via the Meta tag method the tag should be present on each page which is not to be indexed.

<meta name="robots" content="noindex,nofollow">
0
 
LVL 1

Author Closing Comment

by:sahanz
ID: 33455450
Thanks
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
transferring settings for apache and mysql 2 71
file path 14 63
Virtual host in apache 31 73
Redirect and Rewrite URL using .htaccess 38 95
If you've heard about htaccess and it sounds like it does what you want, but you're not sure how it works... well, you're in the right place. Read on. Some Basics #1. It's a file and its filename is .htaccess (yes, with a dot in the front). #…
This code takes an Excel list of URL’s and adds a header titled “URL List”. It then searches through all URL’s in column “A”, looking for duplicates. When a duplicate is found, it is moved to the top of the list. The duplicate URL’s are then highlig…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now