Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions

How to stop Search Engine Indexing my subdirectories and continue indexing their respective Add On Domains

Posted on 2009-05-18
Last Modified: 2012-05-07
i am hosted by bluehost.
I have a number of add-on-domain names such as "example.com" and the files for these domains are kept in subdirectories of my account such as "myaccount.com/example"

i create websites for people and sometimes search engines are finding their way to the subdirectories and subdomains such as "myaccount.com/example" and "example.myaccount.com" instead of simply "example.com"

is there a sure fire way to disallow search engines from indexing these subdirectories and subdomains?

will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).

Will i need some sort of .htaccess script. for example putting this in "myaccount.com/example"
"RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.thedomainyouwanttoredirectto\.com\.au [NC]
RewriteRule (.*) http://www.example.com/directoryyouwanttoredirectfrom$1 [R=301,L]"


Thanks for your help

Question by:simophat
  • 2
  • 2
LVL 23

Expert Comment

by:Tony McCreath
ID: 24420332
If spiders are finding those urls there must be links to them. So the first thing is remove those links.

Your robots.txt file idea will help.

You could also register with Google Webmaster Tools where you can instruct Google to remove links that it has indexed.


Author Comment

ID: 24427863
so will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).

Expert Comment

ID: 24427925
Right Click on the "Google" Logo Under - Part of your Google Toolbar
There is a wrench on the Google toolbar all the way to the right with a drop down arrow (left click)

Google Toolbar Options

All the way at the bottom ...

Uncheck "Enable the Google New Tab Page"

Expert Comment

ID: 24427939
Sorry wrong post - my apologies
LVL 23

Accepted Solution

Tony McCreath earned 500 total points
ID: 24439495
As long as the robots.txt file is in  the root folder of your website then search engines will find it.

The robots.txt file does not seem to work 100%. In particualr it can take some time before pages that are already indexed are removed.

Just to try and clarify your situation

http://myaccount.com  = your main website
http://example.myaccount.com  = an example sub domain
http://myaccount.com/example = also points to the example sub domain

And you want to stop search engines indexing the last one. Then your robots.txt file should be here:


and contain

User-agent: *
Disallow: /example/

Open in new window


Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
seo for images 2 79
Robots.txt and SEO question 5 59
Google Crawl Errors Producing 404 Errors 4 84
Search 5 58
[Part 6 of a 6 part series called SEO Basics: 5 SEO Secrets for Creating Content that Drives Traffic (http://www.experts-exchange.com/Web_Development/Internet_Marketing/Search_Engine_Optimization_SEO/A_8369-SEO-Basics-5-SEO-Secrets-for-Creating-Cont…
Your personal information, where you live, your wedding information, and anything else personal may be located online and in many cases the aggregators who pulled this stuff did so without your permission. Here are ways that you can get rid of unwan…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…

791 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question