Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

How to stop Search Engine Indexing my subdirectories and continue indexing their respective Add On Domains

Posted on 2009-05-18
5
Medium Priority
?
597 Views
Last Modified: 2012-05-07
Hi,
i am hosted by bluehost.
I have a number of add-on-domain names such as "example.com" and the files for these domains are kept in subdirectories of my account such as "myaccount.com/example"

i create websites for people and sometimes search engines are finding their way to the subdirectories and subdomains such as "myaccount.com/example" and "example.myaccount.com" instead of simply "example.com"

is there a sure fire way to disallow search engines from indexing these subdirectories and subdomains?

will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).

Will i need some sort of .htaccess script. for example putting this in "myaccount.com/example"
"RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.thedomainyouwanttoredirectto\.com\.au [NC]
RewriteRule (.*) http://www.example.com/directoryyouwanttoredirectfrom$1 [R=301,L]"

?

Thanks for your help

Simon
0
Comment
Question by:simophat
  • 2
  • 2
5 Comments
 
LVL 23

Expert Comment

by:Tony McCreath
ID: 24420332
If spiders are finding those urls there must be links to them. So the first thing is remove those links.

Your robots.txt file idea will help.

You could also register with Google Webmaster Tools where you can instruct Google to remove links that it has indexed.

https://www.google.com/webmasters/tools
0
 

Author Comment

by:simophat
ID: 24427863
ok,
so will a robot.txt placed in the "myaccount.com/robot.txt" directory, that contains code to disallow search engines from indexing subdirectories (such as "myaccount.com/example") work? Or will that only stop search engines from following links from the root folder (myaccount.com).
0
 
LVL 4

Expert Comment

by:TG_Tech
ID: 24427925
Right Click on the "Google" Logo Under - Part of your Google Toolbar
or
There is a wrench on the Google toolbar all the way to the right with a drop down arrow (left click)

Google Toolbar Options

All the way at the bottom ...

Uncheck "Enable the Google New Tab Page"
0
 
LVL 4

Expert Comment

by:TG_Tech
ID: 24427939
Sorry wrong post - my apologies
0
 
LVL 23

Accepted Solution

by:
Tony McCreath earned 2000 total points
ID: 24439495
As long as the robots.txt file is in  the root folder of your website then search engines will find it.

The robots.txt file does not seem to work 100%. In particualr it can take some time before pages that are already indexed are removed.

Just to try and clarify your situation

http://myaccount.com  = your main website
http://example.myaccount.com  = an example sub domain
http://myaccount.com/example = also points to the example sub domain

And you want to stop search engines indexing the last one. Then your robots.txt file should be here:

http://myaccount.com/robots.txt

and contain

User-agent: *
Disallow: /example/

Open in new window

0

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

With more and more people tethered to the internet, advertisers can reach you in new and creative ways. Push notifications on your smartphone, customized advertisements based on your previous searches, and email inboxes filled with promotional pitch…
SEO can be a real minefield to navigate, but there are three simple ways to up your SEO game just be re-assessing your content output.
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.
Suggested Courses

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question