Solved

How to pevent search engines from finding my sites? Robots.txt not working!

Posted on 2009-03-31
2
222 Views
Last Modified: 2012-05-06
I have a number of websites on a development server that are showing up in Google searches.

I added robots.txt files (see code below)  to the web root of each site months ago...but the sites are still appearing in Google results.

Is there anyway I can hide these sites using Apache? Firewalls? etc?

FYI - All sites are in a folder path like this: /var/www/mysites/...

Any ideas?

# Robots.txt
User-agent: *
Disallow: /

Open in new window

0
Comment
Question by:bearclaws75
2 Comments
 
LVL 27

Accepted Solution

by:
caterham_www earned 250 total points
ID: 24033106
It could take several moths until the sites drop-out of a search engine's index.

> but the sites are still appearing in Google results.

and they're still spidered? Or does, e.g., googlebot request your robots.txt (lower-case r) but not any subsequent files?

You may also submit a removal request to speed things up: http://www.google.com/webmasters/tools/removals

ref: http://www.google.com/support/webmasters/bin/answer.py?answer=61062&ctx=sibling
0
 

Author Comment

by:bearclaws75
ID: 24090788
We added the Robots.txt a few weeks after the pages were originally published so it is likely that pages have not been re-cached.

To be safe, I submitted a removal request using webmaster tools.

Thanks!
0

Featured Post

Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
Because your company can’t afford for you to make SEO mistakes, you’ll want to ensure you’re taking the right steps each and every time you post a new piece of content. This list of optimization do’s and don’ts can help you become an SEO wizard.
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…

806 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question