Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Robot.txt and subdomain

Posted on 2007-12-05
3
Medium Priority
?
642 Views
Last Modified: 2013-12-08
Dear Experts,

I need to ask about robot.txt behaviour.
I have a folder '/var/www/html/images/stories/*'

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/'

will search engine crawl to my subdomain: 'images.mydomain.com/' ?

I want the search engine can crawl into my subdomain but disallow to crawl '/images/*'
is it possible?

or I should make a new pointed folder for my subdomain?

If I make a new pointed folder: /var/www/html/advertImage/
do the search engine get the address 'www.mydomain.com/advertimage/' or 'images.mydomain.com/' ?

Please kindly advise.

Thank you.

Kind Regards,
Aciang.
0
Comment
Question by:aciang
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 

Author Comment

by:aciang
ID: 20412468
ups sorry.
correction on paragraph 2.
it should be:

and my robot.txt disallow folder '/images/'
I make a subdomain: 'images.mydomain.com' that point to '/var/www/html/images/stories/'
0
 
LVL 3

Accepted Solution

by:
hilay earned 2000 total points
ID: 20415153
Crawlers, at least the legit ones, are working for a specific domain as ordered on its robots.txt file.

In your case if you have mydomain.com the crawler will look for robots.txt at the top directory (i.e http://mydomain.com/robots.txt)
This will also be the case for www.mydomain.com (i.e http://mydomain.com/robots.txt)
Keep in mind that those files could, if you like them to be, not be the same.

This is to tell you that sub.mydomain.com could have a different set of rules in its robots.txt the located at http://sub.mydomain.com/robots.txt

So, if you need to hide/show something on a subdomain vs domain you need to alter the robots.txt file but dont leave it without this file.

Good luck
0
 

Author Closing Comment

by:aciang
ID: 31412881
ok got it.
Thank you so much hilay, sorry for the late reply.
Thanks. :)
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When it comes to write a Context Sensitive Help (an online help that is obtained from a specific point in state of software to provide help with that state) ,  first we need to make the file that contains all topics, which are given exclusive IDs. …
When the s#!t hits the fan, you don’t have time to look up who’s on call, draft emails, call collaborators, or send text messages. An instant chat window is definitely the way to go, especially one like HipChat. HipChat is a true business app. An…
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Suggested Courses

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question