Solved

Google SEO: How to provocate a recrawl of a sitemap (update wrong informations)

Posted on 2014-07-25
5
304 Views
Last Modified: 2014-08-19
Hi all

I have a special problem.
We have a public portal created with a software based on AJAX.
As AJAX don't return useful informations to SEO spiders (Google-bot, Bing-bot, etc.) we have implemented a special
"Spider detection" and - if a spider is detected - give back standard html-code with the correct informations to the link (queried from a sql-server).
We have about 38'000 different links.
We have submitted a (generated) site-map with all the links in the Google-WebMaster-Tool.
This has worked fine - our links where indexed fine.

Introduction:
Unfortunately there was a bug in our underlying development-software (with an update), so that the spiders where not detected correct for a longer time. As they don't were detected correct, the standard AJAX-Code was given back for a longer time (no useful information for the spiders).
The result was, that all entries were killed (Google, Bing, Yahoo)...
Nothing was found by do a search, no keywords where stored.
With site:xxxxxx only links (from the sitemap) were displayed (without any further information).
After a long investigation, I have find out the problem and was able to implement a workaround, so that the detection now works correct again.
I then have crawled some of the links manually I the Google-Webmaster-Tool
=> This has worked correct (I was able to find the information's in Google-Search)
I then have re-submitted the site-map (with the about 38'000 Links, same version of sitemap as before).

Problem description:
In the last 3 Weeks maybe 10-20 links (from the about 38'000) were crawled automatically from google.
If I search with site:xxx:
- I can see maybe 60-80 correct (new crawled) entry's - the most of them are from manually submitted links in the
   webmaster-tool.  
- I can see a lot of (wrong) entry's with only the links (without any further information)

Question:
=> How can in solve this problem an force a recrawl of all (about 38'000) links in the sitemap?
==> As I wrote I have submitted the site-map once again without success:
   => But the same version as before (same file-date, same entry's)
 => Maybe I should update the file-date / some contend of the sitemap (does this matter)?

Thanks for any advice...
P.S:
- The same is true in Bing and Yahoo
0
Comment
Question by:Honeymoon
  • 3
  • 2
5 Comments
 
LVL 58

Expert Comment

by:Gary
Comment Utility
force a recrawl
You can't, you just just have to grin and bear it until Google et al eventually reindex the site.
0
 

Author Comment

by:Honeymoon
Comment Utility
Seriously...?
As I can see in the Web-Master-Tool, Google is crawling between 1-20 pages per day (in which I can't see a real progress in search-results day by day).
=> Maybe the thousand's already stored links without any information have something to do with that?

If Google crawls 10 Pages / Day (in average) and I have 38'000 Pages, I have to wait 3'800 days = 10 years?!?

Is there really no other way?
0
 
LVL 58

Expert Comment

by:Gary
Comment Utility
What you see in WMT is not real time.

You have the advantage that the links are still there so it will happen fairly quickly but it's not going to happen in the next week.
You can try increasing the crawl rate in WMT but it doesn't mean Google will honour it if it doesn't think there is any need.
0
 

Author Comment

by:Honeymoon
Comment Utility
Hi Gary

Additional questions:
- Does Google have a look:
  -  at the file-date of the  sitemap?  
  -  at the filesize-date of the  sitemap?  
  -  at the lastmodifyed-attribute in sitemap (to every entry)?

Thanks
Fredy
0
 
LVL 58

Accepted Solution

by:
Gary earned 500 total points
Comment Utility
 -  at the file-date of the  sitemap?  
   -  at the filesize-date of the  sitemap?  
I'm not 100% sure, from vague memory it just downloads it on a somewhat regular basis

-  at the lastmodifyed-attribute in sitemap (to every entry)?
The value makes no difference, Google basically uses the sitemap to learn your site structure/links and nothing more
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

I recently wrote a post (http://blog.freshegg.com/media-importantly-seo_2428) over at Fresh Egg about getting into digital media, but more specifically getting into SEO.  I thought it would be useful to elaborate on that post a little and write abou…
A step by step SEO guide to creating content that drives traffic and maximizes page views by using the right tricks, tools and keyword analysis. You are the subject matter expert.   You have forgotten more about your area of expertise than most …
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now