Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

How to pevent search engines from finding my sites? Robots.txt not working!

Posted on 2009-03-31
2
Medium Priority
?
255 Views
Last Modified: 2012-05-06
I have a number of websites on a development server that are showing up in Google searches.

I added robots.txt files (see code below)  to the web root of each site months ago...but the sites are still appearing in Google results.

Is there anyway I can hide these sites using Apache? Firewalls? etc?

FYI - All sites are in a folder path like this: /var/www/mysites/...

Any ideas?

# Robots.txt
User-agent: *
Disallow: /

Open in new window

0
Comment
Question by:bearclaws75
2 Comments
 
LVL 27

Accepted Solution

by:
caterham_www earned 750 total points
ID: 24033106
It could take several moths until the sites drop-out of a search engine's index.

> but the sites are still appearing in Google results.

and they're still spidered? Or does, e.g., googlebot request your robots.txt (lower-case r) but not any subsequent files?

You may also submit a removal request to speed things up: http://www.google.com/webmasters/tools/removals

ref: http://www.google.com/support/webmasters/bin/answer.py?answer=61062&ctx=sibling
0
 

Author Comment

by:bearclaws75
ID: 24090788
We added the Robots.txt a few weeks after the pages were originally published so it is likely that pages have not been re-cached.

To be safe, I submitted a removal request using webmaster tools.

Thanks!
0

Featured Post

Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Because your company can’t afford for you to make SEO mistakes, you’ll want to ensure you’re taking the right steps each and every time you post a new piece of content. This list of optimization do’s and don’ts can help you become an SEO wizard.
FAQ pages provide a simple way for you to supply and for customers to find answers to the most common questions about your company. Here are six reasons why your company website should have a FAQ page
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.
Suggested Courses
Course of the Month11 days, 23 hours left to enroll

916 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question