Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Virtual Hosting and Robot.txt

Posted on 2006-06-15
6
Medium Priority
?
251 Views
Last Modified: 2010-03-04
I have 2 websites running on my web server.   I am running Linux and Apache.

My question is:  How do I get my robots.txt files working for both domains?

Do I just both a robot.txt in each of the website root directories?  If so, I have done that.   But my second website still is not coming up.  My first website is still showing old stuff for 6 months ago when I search in google for it.  I have updated my site since then with major changes.

But the main one I am concerned about is the second website.   Here is a week, it will be fully functional and I want it to come up in google or other.

Thanks...



0
Comment
Question by:strongd
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 
LVL 9

Expert Comment

by:smidgie82
ID: 16913458
As for your initial question, you need to have robots.txt in the root of each website.  However, robots.txt will in no way help you get indexed faster.  Go to Google Sitemaps and register your sites.  Then download a utility to create sitemaps and submit them.  That WILL help you get indexed faster.
0
 

Author Comment

by:strongd
ID: 16915953
How does the spider know that there is 2 websites or to different directory to go to?
0
 
LVL 9

Accepted Solution

by:
smidgie82 earned 500 total points
ID: 16916141
I think we have a difference of definition here.  By two different websites, I assume you mean two VirtualHost's running on the same or different machine.  In that case, in order to get to either website in the first place, the spider will already need to know the domain name of the website it wants.  It won't know or care that the two are on the same machine, and more than likely it won't be the same spider traversing both sites regardless.  By developing a google sitemap and submitting it to Google, you'll be giving Google's bots a starting address and a list of files you consider important to your site.  That's how they'll find your site, and they'll handle the rest from there.

If the above does not describe your situation (i.e., your two websites share a common name), then please describe your configuration in more detail so I can give appropriate advice.
0
Learn Veeam advantages over legacy backup

Every day, more and more legacy backup customers switch to Veeam. Technologies designed for the client-server era cannot restore any IT service running in the hybrid cloud within seconds. Learn top Veeam advantages over legacy backup and get Veeam for the price of your renewal

 

Author Comment

by:strongd
ID: 16916299
Thanks alot for your help.   Yes, I have 2 websites running on the same machine using virtual hosting.   So I guess I need to go to google and do the sitemap thing.

Thanks again for your help.  I think you answered my question.

0
 
LVL 9

Expert Comment

by:smidgie82
ID: 16916301
My pleasure.  Good luck!
0
 

Author Comment

by:strongd
ID: 16916319
Thanks, hopefully I can get this thing going so I can get people to my second site.   The first site was more or less a play toy/ learning thing...  

Thanks again.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you are running a LAMP infrastructure, this little code snippet is very helpful if you are serving lots of HTML, JavaScript and CSS-related information. The mod_deflate module, which is part of the Apache 2.2 application, provides the DEFLATE…
Introduction This article is intended for those who are new to PHP error handling (https://www.experts-exchange.com/articles/11769/And-by-the-way-I-am-New-to-PHP.html).  It addresses one of the most common problems that plague beginning PHP develop…
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…
In this video, Percona Solution Engineer Dimitri Vanoverbeke discusses why you want to use at least three nodes in a database cluster. To discuss how Percona Consulting can help with your design and architecture needs for your database and infras…
Suggested Courses

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question