Solved

Robot.txt and subdomain

Posted on 2007-12-05
3
638 Views
Last Modified: 2013-12-08
Dear Experts,

I need to ask about robot.txt behaviour.
I have a folder '/var/www/html/images/stories/*'

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/'

will search engine crawl to my subdomain: 'images.mydomain.com/' ?

I want the search engine can crawl into my subdomain but disallow to crawl '/images/*'
is it possible?

or I should make a new pointed folder for my subdomain?

If I make a new pointed folder: /var/www/html/advertImage/
do the search engine get the address 'www.mydomain.com/advertimage/' or 'images.mydomain.com/' ?

Please kindly advise.

Thank you.

Kind Regards,
Aciang.
0
Comment
Question by:aciang
  • 2
3 Comments
 

Author Comment

by:aciang
ID: 20412468
ups sorry.
correction on paragraph 2.
it should be:

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/stories/'
0
 
LVL 3

Accepted Solution

by:
hilay earned 500 total points
ID: 20415153
Crawlers, at least the legit ones, are working for a specific domain as ordered on its robots.txt file.

In your case if you have mydomain.com the crawler will look for robots.txt at the top directory (i.e http://mydomain.com/robots.txt)
This will also be the case for www.mydomain.com (i.e http://mydomain.com/robots.txt)
Keep in mind that those files could, if you like them to be, not be the same.

This is to tell you that sub.mydomain.com could have a different set of rules in its robots.txt the located at http://sub.mydomain.com/robots.txt

So, if you need to hide/show something on a subdomain vs domain you need to alter the robots.txt file but dont leave it without this file.

Good luck
0
 

Author Closing Comment

by:aciang
ID: 31412881
ok got it.
Thank you so much hilay, sorry for the late reply.
Thanks. :)
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this blog, I will share you some basic tips for content marketing and to rank your website on Google.
FAQ pages provide a simple way for you to supply and for customers to find answers to the most common questions about your company. Here are six reasons why your company website should have a FAQ page
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.

820 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question