Solved

Robot.txt and subdomain

Posted on 2007-12-05
3
633 Views
Last Modified: 2013-12-08
Dear Experts,

I need to ask about robot.txt behaviour.
I have a folder '/var/www/html/images/stories/*'

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/'

will search engine crawl to my subdomain: 'images.mydomain.com/' ?

I want the search engine can crawl into my subdomain but disallow to crawl '/images/*'
is it possible?

or I should make a new pointed folder for my subdomain?

If I make a new pointed folder: /var/www/html/advertImage/
do the search engine get the address 'www.mydomain.com/advertimage/' or 'images.mydomain.com/' ?

Please kindly advise.

Thank you.

Kind Regards,
Aciang.
0
Comment
Question by:aciang
  • 2
3 Comments
 

Author Comment

by:aciang
ID: 20412468
ups sorry.
correction on paragraph 2.
it should be:

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/stories/'
0
 
LVL 3

Accepted Solution

by:
hilay earned 500 total points
ID: 20415153
Crawlers, at least the legit ones, are working for a specific domain as ordered on its robots.txt file.

In your case if you have mydomain.com the crawler will look for robots.txt at the top directory (i.e http://mydomain.com/robots.txt)
This will also be the case for www.mydomain.com (i.e http://mydomain.com/robots.txt)
Keep in mind that those files could, if you like them to be, not be the same.

This is to tell you that sub.mydomain.com could have a different set of rules in its robots.txt the located at http://sub.mydomain.com/robots.txt

So, if you need to hide/show something on a subdomain vs domain you need to alter the robots.txt file but dont leave it without this file.

Good luck
0
 

Author Closing Comment

by:aciang
ID: 31412881
ok got it.
Thank you so much hilay, sorry for the late reply.
Thanks. :)
0

Featured Post

DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
With more and more people tethered to the internet, advertisers can reach you in new and creative ways. Push notifications on your smartphone, customized advertisements based on your previous searches, and email inboxes filled with promotional pitch…
This video teaches users how to migrate an existing Wordpress website to a new domain.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question