I have a special problem.
We have a public portal created with a software based on AJAX.
As AJAX don't return useful informations to SEO spiders (Google-bot, Bing-bot, etc.) we have implemented a special
"Spider detection" and - if a spider is detected - give back standard html-code with the correct informations to the link (queried from a sql-server).
We have about 38'000 different links.
We have submitted a (generated) site-map with all the links in the Google-WebMaster-Tool.
This has worked fine - our links where indexed fine.
Unfortunately there was a bug in our underlying development-software (with an update), so that the spiders where not detected correct for a longer time. As they don't were detected correct, the standard AJAX-Code was given back for a longer time (no useful information for the spiders).
The result was, that all entries were killed (Google, Bing, Yahoo)...
Nothing was found by do a search, no keywords where stored.
With site:xxxxxx only links (from the sitemap) were displayed (without any further information).
After a long investigation, I have find out the problem and was able to implement a workaround, so that the detection now works correct again.
I then have crawled some of the links manually I the Google-Webmaster-Tool
=> This has worked correct (I was able to find the information's in Google-Search)
I then have re-submitted the site-map (with the about 38'000 Links, same version of sitemap as before).
In the last 3 Weeks maybe 10-20 links (from the about 38'000) were crawled automatically from google.
If I search with site:xxx:
- I can see maybe 60-80 correct (new crawled) entry's - the most of them are from manually submitted links in the
- I can see a lot of (wrong) entry's with only the links (without any further information)
=> How can in solve this problem an force a recrawl of all (about 38'000) links in the sitemap?
==> As I wrote I have submitted the site-map once again without success:
=> But the same version as before (same file-date, same entry's)
=> Maybe I should update the file-date / some contend of the sitemap (does this matter)?
Thanks for any advice...
- The same is true in Bing and Yahoo