Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Robot.txt and subdomain

Posted on 2007-12-05
3
Medium Priority
?
645 Views
Last Modified: 2013-12-08
Dear Experts,

I need to ask about robot.txt behaviour.
I have a folder '/var/www/html/images/stories/*'

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/'

will search engine crawl to my subdomain: 'images.mydomain.com/' ?

I want the search engine can crawl into my subdomain but disallow to crawl '/images/*'
is it possible?

or I should make a new pointed folder for my subdomain?

If I make a new pointed folder: /var/www/html/advertImage/
do the search engine get the address 'www.mydomain.com/advertimage/' or 'images.mydomain.com/' ?

Please kindly advise.

Thank you.

Kind Regards,
Aciang.
0
Comment
Question by:aciang
  • 2
3 Comments
 

Author Comment

by:aciang
ID: 20412468
ups sorry.
correction on paragraph 2.
it should be:

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/stories/'
0
 
LVL 3

Accepted Solution

by:
hilay earned 2000 total points
ID: 20415153
Crawlers, at least the legit ones, are working for a specific domain as ordered on its robots.txt file.

In your case if you have mydomain.com the crawler will look for robots.txt at the top directory (i.e http://mydomain.com/robots.txt)
This will also be the case for www.mydomain.com (i.e http://mydomain.com/robots.txt)
Keep in mind that those files could, if you like them to be, not be the same.

This is to tell you that sub.mydomain.com could have a different set of rules in its robots.txt the located at http://sub.mydomain.com/robots.txt

So, if you need to hide/show something on a subdomain vs domain you need to alter the robots.txt file but dont leave it without this file.

Good luck
0
 

Author Closing Comment

by:aciang
ID: 31412881
ok got it.
Thank you so much hilay, sorry for the late reply.
Thanks. :)
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When crafting your “Why Us” page, there are a plethora of pitfalls to avoid. Follow these five tips, and you’ll be well on your way to creating an effective page.
Ready to get certified? Check out some courses that help you prepare for third-party exams.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
Suggested Courses

972 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question