Solved

Google SEO: How to provocate a recrawl of a sitemap (update wrong informations)

Posted on 2014-07-25
5
313 Views
Last Modified: 2014-08-19
Hi all

I have a special problem.
We have a public portal created with a software based on AJAX.
As AJAX don't return useful informations to SEO spiders (Google-bot, Bing-bot, etc.) we have implemented a special
"Spider detection" and - if a spider is detected - give back standard html-code with the correct informations to the link (queried from a sql-server).
We have about 38'000 different links.
We have submitted a (generated) site-map with all the links in the Google-WebMaster-Tool.
This has worked fine - our links where indexed fine.

Introduction:
Unfortunately there was a bug in our underlying development-software (with an update), so that the spiders where not detected correct for a longer time. As they don't were detected correct, the standard AJAX-Code was given back for a longer time (no useful information for the spiders).
The result was, that all entries were killed (Google, Bing, Yahoo)...
Nothing was found by do a search, no keywords where stored.
With site:xxxxxx only links (from the sitemap) were displayed (without any further information).
After a long investigation, I have find out the problem and was able to implement a workaround, so that the detection now works correct again.
I then have crawled some of the links manually I the Google-Webmaster-Tool
=> This has worked correct (I was able to find the information's in Google-Search)
I then have re-submitted the site-map (with the about 38'000 Links, same version of sitemap as before).

Problem description:
In the last 3 Weeks maybe 10-20 links (from the about 38'000) were crawled automatically from google.
If I search with site:xxx:
- I can see maybe 60-80 correct (new crawled) entry's - the most of them are from manually submitted links in the
   webmaster-tool.  
- I can see a lot of (wrong) entry's with only the links (without any further information)

Question:
=> How can in solve this problem an force a recrawl of all (about 38'000) links in the sitemap?
==> As I wrote I have submitted the site-map once again without success:
   => But the same version as before (same file-date, same entry's)
 => Maybe I should update the file-date / some contend of the sitemap (does this matter)?

Thanks for any advice...
P.S:
- The same is true in Bing and Yahoo
0
Comment
Question by:Honeymoon
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
5 Comments
 
LVL 58

Expert Comment

by:Gary
ID: 40219381
force a recrawl
You can't, you just just have to grin and bear it until Google et al eventually reindex the site.
0
 

Author Comment

by:Honeymoon
ID: 40219764
Seriously...?
As I can see in the Web-Master-Tool, Google is crawling between 1-20 pages per day (in which I can't see a real progress in search-results day by day).
=> Maybe the thousand's already stored links without any information have something to do with that?

If Google crawls 10 Pages / Day (in average) and I have 38'000 Pages, I have to wait 3'800 days = 10 years?!?

Is there really no other way?
0
 
LVL 58

Expert Comment

by:Gary
ID: 40219781
What you see in WMT is not real time.

You have the advantage that the links are still there so it will happen fairly quickly but it's not going to happen in the next week.
You can try increasing the crawl rate in WMT but it doesn't mean Google will honour it if it doesn't think there is any need.
0
 

Author Comment

by:Honeymoon
ID: 40252603
Hi Gary

Additional questions:
- Does Google have a look:
  -  at the file-date of the  sitemap?  
  -  at the filesize-date of the  sitemap?  
  -  at the lastmodifyed-attribute in sitemap (to every entry)?

Thanks
Fredy
0
 
LVL 58

Accepted Solution

by:
Gary earned 500 total points
ID: 40253154
 -  at the file-date of the  sitemap?  
   -  at the filesize-date of the  sitemap?  
I'm not 100% sure, from vague memory it just downloads it on a somewhat regular basis

-  at the lastmodifyed-attribute in sitemap (to every entry)?
The value makes no difference, Google basically uses the sitemap to learn your site structure/links and nothing more
0

Featured Post

Instantly Create Instructional Tutorials

Contextual Guidance at the moment of need helps your employees adopt to new software or processes instantly. Boost knowledge retention and employee engagement step-by-step with one easy solution.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A/B testing is a simple and effective trick to get to know your audience, increase website conversions and make the most out of your online ad campaigns. It's widely available and doesn't need much tech knowledge to be executed, but the results it y…
SEO can be a real minefield to navigate, but there are three simple ways to up your SEO game just be re-assessing your content output.
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…

729 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question