How to stop Search Engine Indexing my subdirectories and continue indexing their respective Add On Domains

Hi,
i am hosted by bluehost.
I have a number of add-on-domain names such as "example.com" and the files for these domains are kept in subdirectories of my account such as "myaccount.com/example"

i create websites for people and sometimes search engines are finding their way to the subdirectories and subdomains such as "myaccount.com/example" and "example.myaccount.com" instead of simply "example.com"

is there a sure fire way to disallow search engines from indexing these subdirectories and subdomains?

will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).

Will i need some sort of .htaccess script. for example putting this in "myaccount.com/example"
"RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.thedomainyouwanttoredirectto\.com\.au [NC]
RewriteRule (.*) http://www.example.com/directoryyouwanttoredirectfrom$1 [R=301,L]"

?

Thanks for your help

Simon
simophatAsked:
Who is Participating?
 
Tony McCreathTechnical SEO ConsultantCommented:
As long as the robots.txt file is in  the root folder of your website then search engines will find it.

The robots.txt file does not seem to work 100%. In particualr it can take some time before pages that are already indexed are removed.

Just to try and clarify your situation

http://myaccount.com  = your main website
http://example.myaccount.com  = an example sub domain
http://myaccount.com/example = also points to the example sub domain

And you want to stop search engines indexing the last one. Then your robots.txt file should be here:

http://myaccount.com/robots.txt

and contain

User-agent: *
Disallow: /example/

Open in new window

0
 
Tony McCreathTechnical SEO ConsultantCommented:
If spiders are finding those urls there must be links to them. So the first thing is remove those links.

Your robots.txt file idea will help.

You could also register with Google Webmaster Tools where you can instruct Google to remove links that it has indexed.

https://www.google.com/webmasters/tools
0
 
simophatAuthor Commented:
ok,
so will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).
0
 
TG_TechCommented:
Right Click on the "Google" Logo Under - Part of your Google Toolbar
or
There is a wrench on the Google toolbar all the way to the right with a drop down arrow (left click)

Google Toolbar Options

All the way at the bottom ...

Uncheck "Enable the Google New Tab Page"
0
 
TG_TechCommented:
Sorry wrong post - my apologies
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.