Solved

How to stop Search Engine Indexing my subdirectories and continue indexing their respective Add On Domains

Posted on 2009-05-18
5
584 Views
Last Modified: 2012-05-07
Hi,
i am hosted by bluehost.
I have a number of add-on-domain names such as "example.com" and the files for these domains are kept in subdirectories of my account such as "myaccount.com/example"

i create websites for people and sometimes search engines are finding their way to the subdirectories and subdomains such as "myaccount.com/example" and "example.myaccount.com" instead of simply "example.com"

is there a sure fire way to disallow search engines from indexing these subdirectories and subdomains?

will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).

Will i need some sort of .htaccess script. for example putting this in "myaccount.com/example"
"RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.thedomainyouwanttoredirectto\.com\.au [NC]
RewriteRule (.*) http://www.example.com/directoryyouwanttoredirectfrom$1 [R=301,L]"

?

Thanks for your help

Simon
0
Comment
Question by:simophat
  • 2
  • 2
5 Comments
 
LVL 23

Expert Comment

by:Tiggerito
ID: 24420332
If spiders are finding those urls there must be links to them. So the first thing is remove those links.

Your robots.txt file idea will help.

You could also register with Google Webmaster Tools where you can instruct Google to remove links that it has indexed.

https://www.google.com/webmasters/tools
0
 

Author Comment

by:simophat
ID: 24427863
ok,
so will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).
0
 
LVL 4

Expert Comment

by:TG_Tech
ID: 24427925
Right Click on the "Google" Logo Under - Part of your Google Toolbar
or
There is a wrench on the Google toolbar all the way to the right with a drop down arrow (left click)

Google Toolbar Options

All the way at the bottom ...

Uncheck "Enable the Google New Tab Page"
0
 
LVL 4

Expert Comment

by:TG_Tech
ID: 24427939
Sorry wrong post - my apologies
0
 
LVL 23

Accepted Solution

by:
Tiggerito earned 500 total points
ID: 24439495
As long as the robots.txt file is in  the root folder of your website then search engines will find it.

The robots.txt file does not seem to work 100%. In particualr it can take some time before pages that are already indexed are removed.

Just to try and clarify your situation

http://myaccount.com  = your main website
http://example.myaccount.com  = an example sub domain
http://myaccount.com/example = also points to the example sub domain

And you want to stop search engines indexing the last one. Then your robots.txt file should be here:

http://myaccount.com/robots.txt

and contain

User-agent: *

Disallow: /example/

Open in new window

0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

I recently wrote a post (http://blog.freshegg.com/media-importantly-seo_2428) over at Fresh Egg about getting into digital media, but more specifically getting into SEO.  I thought it would be useful to elaborate on that post a little and write abou…
A/B testing is a simple and effective trick to get to know your audience, increase website conversions and make the most out of your online ad campaigns. It's widely available and doesn't need much tech knowledge to be executed, but the results it y…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now