Solved

Google to get results from internal database

Posted on 2004-08-11
7
299 Views
Last Modified: 2007-02-13
Hi There,

I am developing a site that has loads of products in a database [stored on our dedicated server]
The client has requested that the products be returned when a user does a google search... I know this is posible as I have seen it done on many sites [including this one] i.e. if I where to go to google and search for "Google to get results from internal database" it would return this page.... somehow.....

Does it have to do with the way the site is developed?
Databse setup? Special Submission to the search engines?

Any help will be greatly appreciated.
0
Comment
Question by:d2kagw
7 Comments
 
LVL 3

Expert Comment

by:passmark
ID: 11779786

Search engine spiders follow links that you have on your web site. The scans web pages as if it was a normal user browsing your site. Provided all your database content is accessible via web pages (ie. there is a URL to each record in your database), the spider mode indexing would work, regardless of the database backend you are using (MySQL, Postgres, MS Access, Oracle, etc.).

However, if there are portions of your database which can not be accessed via your web pages, then you may need to create a list of URLs to refer to these records in your database.

For example, you may have a database of cooking recipe. Each of these recipe are a record in the database, and are accessed via a PHP webpage which displays the recipe selected such as the following:

http://mysite.com/recipe.php?id=12
http://mysite.com/recipe.php?name=pavalova
etc.

We develop our own search engine spider software (for searching within a website) and give this advice to our customers, see here,
http://www.wrensoft.com/zoom/support.html#sql

David
0
 
LVL 1

Expert Comment

by:gvmdevelopment
ID: 11780372
Many search engines will not index dynamic pages (.php, .asp, cgi, etc or URLs that have ? in them) The best solution to get these type of pages indexed into search engines is to output them into .html pages and create links to them. These static pages would only be the entry point into the site and then everything else would be dynamic. It is then up to you how often you create or update these static pages.

You can either write an application to create these pages, or there are applications available which will do the dynamic to static page creation for you on a regular basis.
0
 
LVL 3

Accepted Solution

by:
passmark earned 400 total points
ID: 11780459

> Many search engines will not index dynamic pages...

Well maybe. I don't know of any however.

This is not correct. Google *does* index dymanic pages. Doing a search for just about any topic in Google returns lots of dymanic pages. Google also indexes URLs with a '?'.

For example, doing a Google search for "Highlighters Pencils Pens", returns this link as number #1
www.4imprint.com/Products/ViewSuperGroup.aspx?sgrp=2
and this link as #3
http://eshop.msn.com/category.aspx?catId=987

Both are dymanic database driven sites with '?' in their URLs.

Maybe what gvmdevelopment means is that that these type of dymanic URLs are not always ranked as highly as other 'plain' URLs in some search engines. But this is often becuase the key words don't appear in the URL.

For example,
www.mysite.com/Highlighters-Pencils-Pens.htm

is going to look more relevent to search engine than

www.mysite.com/product.php?ProductId=123

David
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 1

Expert Comment

by:gvmdevelopment
ID: 11780502
You are correct, Google does index these pages, but in many cases it looks down on such pages. The most important part like David said is that there be links to each and every product page, so that the search engine can find them.
0
 
LVL 3

Author Comment

by:d2kagw
ID: 11780699
Good Stuff, two more questions then:
[I have increased points to 400 too]

01.
Do you guys know of a good spider/robot that I could use to test the site [I need to make sure that particular areas are definatly found and others not found at all]

02.
David, you mentioned "creating a list of URLs to refer to these records in your database", would this mean just making a page that is filled with links like:
www.mysite.com/product.php?ProductId=123
www.mysite.com/product.php?ProductId=124
www.mysite.com/product.php?ProductId=125
www.mysite.com/product.php?ProductId=126
www.mysite.com/product.php?ProductId=127
if that was the case, how would you get the spider/robot to look at that page, without it being a hard link?

Thanks again guys !
0
 
LVL 3

Expert Comment

by:passmark
ID: 11780952

1/
My company sells the Zoom Search Engine that you could use to spider you site. But it is only free for small sites.
See, http://www.wrensoft.com and http://www.wrensoft.com/zoom/screen1.html
You just point it at your home page and hit the start button.

Otherwise there are some free alternatives on the web which may not be as easy to use (in my rather bias opinion).

2/
Yes. But here is your real problem.
Just because Google *could* index your site, doesn't mean it will. You will have the same problem as many other web designers. Namely, how do you optimize your site to get good placement (or even any placement at all) in Google? Google might also penalise or ignore pages that look like they have no significant content. So I don't think a straight list of links is going to look very interesting for Google. Whole books (and websites) have been written about placement optimization. Do a search for "Search Engine Optimization".

David
0
 
LVL 24

Expert Comment

by:duz
ID: 11781421
d2kagw -

>Do a search for "Search Engine Optimization".

Don't do that because 99% of what you will read will be complete rubbish or out of date information. Search the "Online Marketing" channel here on Experts Exchange for more reliable and up to date answers.

- duz
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Foreword (July, 2015) Since I first wrote this article, years ago, a great many more people have begun using the internet.  They are coming online from every part of the globe, learning, reading, shopping and spending money at an ever-increasing ra…
I've been asked to discuss some of the UX activities that I'm using with my team. Here I will share some details about how we approach UX projects.
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
The viewer will get a basic understanding of what section 508 compliance can entail, learn about skip navigation links, alt text, transcripts, and font size controls.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now