Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 253
  • Last Modified:

Virtual Hosting and Robot.txt

I have 2 websites running on my web server.   I am running Linux and Apache.

My question is:  How do I get my robots.txt files working for both domains?

Do I just both a robot.txt in each of the website root directories?  If so, I have done that.   But my second website still is not coming up.  My first website is still showing old stuff for 6 months ago when I search in google for it.  I have updated my site since then with major changes.

But the main one I am concerned about is the second website.   Here is a week, it will be fully functional and I want it to come up in google or other.

Thanks...



0
strongd
Asked:
strongd
  • 3
  • 3
1 Solution
 
smidgie82Commented:
As for your initial question, you need to have robots.txt in the root of each website.  However, robots.txt will in no way help you get indexed faster.  Go to Google Sitemaps and register your sites.  Then download a utility to create sitemaps and submit them.  That WILL help you get indexed faster.
0
 
strongdAuthor Commented:
How does the spider know that there is 2 websites or to different directory to go to?
0
 
smidgie82Commented:
I think we have a difference of definition here.  By two different websites, I assume you mean two VirtualHost's running on the same or different machine.  In that case, in order to get to either website in the first place, the spider will already need to know the domain name of the website it wants.  It won't know or care that the two are on the same machine, and more than likely it won't be the same spider traversing both sites regardless.  By developing a google sitemap and submitting it to Google, you'll be giving Google's bots a starting address and a list of files you consider important to your site.  That's how they'll find your site, and they'll handle the rest from there.

If the above does not describe your situation (i.e., your two websites share a common name), then please describe your configuration in more detail so I can give appropriate advice.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
strongdAuthor Commented:
Thanks alot for your help.   Yes, I have 2 websites running on the same machine using virtual hosting.   So I guess I need to go to google and do the sitemap thing.

Thanks again for your help.  I think you answered my question.

0
 
smidgie82Commented:
My pleasure.  Good luck!
0
 
strongdAuthor Commented:
Thanks, hopefully I can get this thing going so I can get people to my second site.   The first site was more or less a play toy/ learning thing...  

Thanks again.
0

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

  • 3
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now