Robot.txt and subdomain

Dear Experts,

I need to ask about robot.txt behaviour.
I have a folder '/var/www/html/images/stories/*'

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/'

will search engine crawl to my subdomain: 'images.mydomain.com/' ?

I want the search engine can crawl into my subdomain but disallow to crawl '/images/*'
is it possible?

or I should make a new pointed folder for my subdomain?

If I make a new pointed folder: /var/www/html/advertImage/
do the search engine get the address 'www.mydomain.com/advertimage/' or 'images.mydomain.com/' ?

Please kindly advise.

Thank you.

Kind Regards,
Aciang.
aciangAsked:
Who is Participating?
 
hilayConnect With a Mentor Commented:
Crawlers, at least the legit ones, are working for a specific domain as ordered on its robots.txt file.

In your case if you have mydomain.com the crawler will look for robots.txt at the top directory (i.e http://mydomain.com/robots.txt)
This will also be the case for www.mydomain.com (i.e http://mydomain.com/robots.txt)
Keep in mind that those files could, if you like them to be, not be the same.

This is to tell you that sub.mydomain.com could have a different set of rules in its robots.txt the located at http://sub.mydomain.com/robots.txt

So, if you need to hide/show something on a subdomain vs domain you need to alter the robots.txt file but dont leave it without this file.

Good luck
0
 
aciangAuthor Commented:
ups sorry.
correction on paragraph 2.
it should be:

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/stories/'
0
 
aciangAuthor Commented:
ok got it.
Thank you so much hilay, sorry for the late reply.
Thanks. :)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.