?
Solved

GoogleBot shoes site unreadable (500)

Posted on 2010-08-15
3
Medium Priority
?
555 Views
Last Modified: 2013-12-09
Hello all,

Over a week ago we switched our site from ASP to ASP.NET, and 2 days ago I discovered that Google has not been able to index our site.  In Webmaster tools both the site map XML and the googlebot are reporting site unreachable.  

I checked the robot.tx and Google is able to see that, I even have it now with no disable links. When I try the fetch as googlebot, it shows unreadable.  I put a html page into the fetch and it finds it fine.

I looked into my web logs and I see the 500 error happening when teh browser is the googlebot.

I found web posts about the issue with the ASP.net 2.0 Mozilla detection hole and followed those steps.  Do not think that is the issues though. I loaded the plugging into FF (User Agent Switcher) and I can browse the site with no errors.  

I am not sure what else I am missing or where else to check.

My site is www.planetshoes.com

Any help would be great.
0
Comment
Question by:planet_scott
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 20

Expert Comment

by:edster9999
ID: 33441771
The DNS entries are wrong.

On my server I did a dig command to see how it resolved and it worked ok.  It had 2 CNAME entries which is not exactly good practice but should work.
The top level host (normally the company you registered the name with) was akamaiedge.net

I then tried it on a web based version and this went to different top level DNS hosts

http://mydnscheck.com/index.php?op=diagnose&domain=www.planetshoes.com&ns1=&ns2=&ns3=&ns4=&ns5=&ns6=

Nameservers for PLANETSHOES.COM are as follows:
NS89.WORLDNIC.COM
NS90.WORLDNIC.COM

and this fails at the end with no nameservers for the www. bit

So first step is to check the NS records for PLANETSHOES.COM.  This should point to 2 (or more) servers that can tell anyone where the rest of the servers are for this domain.
0
 

Author Comment

by:planet_scott
ID: 33442081
Nothing has changed on our DNS in 6 months.  We use Akamai for CDN.  Google had no issue with our DNS before the move to ASP.net.

I will look into the findings above, but This can not be the issue, since Google can find our site to load the sitemap.  It can find the robot.txt and also I can fetch as Google, any page on the server accept the .net pages.

I even put an classic ASP page back on the site and I was able to do a fetch by Googlebot.

So I think this issue is internal to my web server.
0
 

Accepted Solution

by:
planet_scott earned 0 total points
ID: 33443032
I might have figured out the issue. We have the following code:
 User_Prefer_Langs = System.Web.HttpContext.Current.Request.ServerVariables["HTTP_ACCEPT_LANGUAGE"].ToString();  
arrLang = User_Prefer_Langs.Split(',');
User_Prefer_Lang = arrLang[0];   // always get the first language

This might have been the issue.  I was trying to get the language the browser is set to.
We changed to just set to "EN".  Looks like it might have fixed the issues.
I hope to see soon.
0

Featured Post

Get real performance insights from real users

Key features:
- Total Pages Views and Load times
- Top Pages Viewed and Load Times
- Real Time Site Page Build Performance
- Users’ Browser and Platform Performance
- Geographic User Breakdown
- And more

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Does your audience prefer people in photos or no people? How can you best highlight what you’re selling? What are your competitors doing, and what can you do that is different and unique from them?  Continue reading to learn how to make your images …
When crafting your “Why Us” page, there are a plethora of pitfalls to avoid. Follow these five tips, and you’ll be well on your way to creating an effective page.
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…
The viewer will get a basic understanding of what section 508 compliance can entail, learn about skip navigation links, alt text, transcripts, and font size controls.
Suggested Courses

741 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question