How to stop Search Engine Indexing my subdirectories and continue indexing their respective Add On Domains

Posted on 2009-05-18
Last Modified: 2012-05-07
i am hosted by bluehost.
I have a number of add-on-domain names such as "" and the files for these domains are kept in subdirectories of my account such as ""

i create websites for people and sometimes search engines are finding their way to the subdirectories and subdomains such as "" and "" instead of simply ""

is there a sure fire way to disallow search engines from indexing these subdirectories and subdomains?

will a robot.txt placed in the "" directory, that contains code to disallow search engines from indexing subdirectories (such as "") work? Or will that only stop search engines from following links from the root folder (

Will i need some sort of .htaccess script. for example putting this in ""
"RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.thedomainyouwanttoredirectto\.com\.au [NC]
RewriteRule (.*)$1 [R=301,L]"


Thanks for your help

Question by:simophat
  • 2
  • 2
LVL 23

Expert Comment

ID: 24420332
If spiders are finding those urls there must be links to them. So the first thing is remove those links.

Your robots.txt file idea will help.

You could also register with Google Webmaster Tools where you can instruct Google to remove links that it has indexed.

Author Comment

ID: 24427863
so will a robot.txt placed in the "" directory, that contains code to disallow search engines from indexing subdirectories (such as "") work? Or will that only stop search engines from following links from the root folder (

Expert Comment

ID: 24427925
Right Click on the "Google" Logo Under - Part of your Google Toolbar
There is a wrench on the Google toolbar all the way to the right with a drop down arrow (left click)

Google Toolbar Options

All the way at the bottom ...

Uncheck "Enable the Google New Tab Page"

Expert Comment

ID: 24427939
Sorry wrong post - my apologies
LVL 23

Accepted Solution

Tiggerito earned 500 total points
ID: 24439495
As long as the robots.txt file is in  the root folder of your website then search engines will find it.

The robots.txt file does not seem to work 100%. In particualr it can take some time before pages that are already indexed are removed.

Just to try and clarify your situation  = your main website  = an example sub domain = also points to the example sub domain

And you want to stop search engines indexing the last one. Then your robots.txt file should be here:

and contain

User-agent: *

Disallow: /example/

Open in new window


Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Outlook 2016 Search Error 13 2,922
seo strategy 1 421
How will we SEO work if our blog posts are pointed to our primary domain knowledge base? 8 133
SEO Help. 4 97
A step by step SEO guide to creating content that drives traffic and maximizes page views by using the right tricks, tools and keyword analysis. You are the subject matter expert.   You have forgotten more about your area of expertise than most …
[Part 6 of a 6 part series called SEO Basics: 5 SEO Secrets for Creating Content that Drives Traffic (…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

26 Experts available now in Live!

Get 1:1 Help Now