bing and google

I have a website that has over 5000 links generated on the sitemap. Half are products and the other half are images. Effectively there is one image per product code.
The search engines are ignoring almost all of these links and just indexing 70 pages.
Where can I find some good guidelines on how the search engines operate. When I posted a message on google all I received was a comment 'how many pages do you expect to be indexed!'
michaeldean99Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Dave BaldwinFixer of ProblemsCommented:
Maybe more importantly, Why would they index the other 4930 links?  Are they popular?  Are there external links to them?  Do they match any search requests?

Google doesn't index all of the links in any of the sites I've done.
Julian MatzTechnical SupportCommented:
A sitemap will inform Google of the pages, but they don't guarantee that pages will be "picked up". The first question I would ask is: do the pages have unique content, or are each of the pages very similar in content? Google and others don't seem to like duplicate content. This includes meta tags and page titles.

Also, I would make sure you follow the quality guidelines.

Have a look at these:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35769
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=66359

My own number 1 rule is to create content for users and not for search engines. Hence, I prefer to use the term HEO (Human Experience Optimisation) rather than SEO (Search Engine Optimisation).
michaeldean99Author Commented:
Why would they index the other 4930 links?


Half are product images, they are of almost all the UK stamps issued from day one! Google wants more images and these are unique images of each issued postage stamp.
But it the product codes themselves that I am trying to get indexed. People around the world type in the standard code for each of these stamps because they are looking to buy them or least check the prices.
Before last Christmas they were mostly indexed . I changed the hosting at  Christmas and the site was down for a few weeks and now I cannot get any of them listed.
It looks as though Google is considering the volume of items to be an attempt to manipulate the search engine. In reality they are all unique and separate items in which (many) people are interested.
Dave BaldwinFixer of ProblemsCommented:
Have you submitted a sitemap in Google Webmaster Tools?  And do other people link to your pages?  And what happens when you search for one of the non-indexed but rare links or products?

Google frequently adjusts their methods of listing and indexing pages.  Since they are probably the world's largest advertising company now, I think that drives their viewpoint.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
ASP

From novice to tech pro — start learning today.