Solved

Google to get results from internal database

Posted on 2004-08-11
7
305 Views
Last Modified: 2007-02-13
Hi There,

I am developing a site that has loads of products in a database [stored on our dedicated server]
The client has requested that the products be returned when a user does a google search... I know this is posible as I have seen it done on many sites [including this one] i.e. if I where to go to google and search for "Google to get results from internal database" it would return this page.... somehow.....

Does it have to do with the way the site is developed?
Databse setup? Special Submission to the search engines?

Any help will be greatly appreciated.
0
Comment
Question by:d2kagw
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
7 Comments
 
LVL 3

Expert Comment

by:passmark
ID: 11779786

Search engine spiders follow links that you have on your web site. The scans web pages as if it was a normal user browsing your site. Provided all your database content is accessible via web pages (ie. there is a URL to each record in your database), the spider mode indexing would work, regardless of the database backend you are using (MySQL, Postgres, MS Access, Oracle, etc.).

However, if there are portions of your database which can not be accessed via your web pages, then you may need to create a list of URLs to refer to these records in your database.

For example, you may have a database of cooking recipe. Each of these recipe are a record in the database, and are accessed via a PHP webpage which displays the recipe selected such as the following:

http://mysite.com/recipe.php?id=12
http://mysite.com/recipe.php?name=pavalova
etc.

We develop our own search engine spider software (for searching within a website) and give this advice to our customers, see here,
http://www.wrensoft.com/zoom/support.html#sql

David
0
 
LVL 1

Expert Comment

by:gvmdevelopment
ID: 11780372
Many search engines will not index dynamic pages (.php, .asp, cgi, etc or URLs that have ? in them) The best solution to get these type of pages indexed into search engines is to output them into .html pages and create links to them. These static pages would only be the entry point into the site and then everything else would be dynamic. It is then up to you how often you create or update these static pages.

You can either write an application to create these pages, or there are applications available which will do the dynamic to static page creation for you on a regular basis.
0
 
LVL 3

Accepted Solution

by:
passmark earned 400 total points
ID: 11780459

> Many search engines will not index dynamic pages...

Well maybe. I don't know of any however.

This is not correct. Google *does* index dymanic pages. Doing a search for just about any topic in Google returns lots of dymanic pages. Google also indexes URLs with a '?'.

For example, doing a Google search for "Highlighters Pencils Pens", returns this link as number #1
www.4imprint.com/Products/ViewSuperGroup.aspx?sgrp=2
and this link as #3
http://eshop.msn.com/category.aspx?catId=987

Both are dymanic database driven sites with '?' in their URLs.

Maybe what gvmdevelopment means is that that these type of dymanic URLs are not always ranked as highly as other 'plain' URLs in some search engines. But this is often becuase the key words don't appear in the URL.

For example,
www.mysite.com/Highlighters-Pencils-Pens.htm

is going to look more relevent to search engine than

www.mysite.com/product.php?ProductId=123

David
0
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

 
LVL 1

Expert Comment

by:gvmdevelopment
ID: 11780502
You are correct, Google does index these pages, but in many cases it looks down on such pages. The most important part like David said is that there be links to each and every product page, so that the search engine can find them.
0
 
LVL 3

Author Comment

by:d2kagw
ID: 11780699
Good Stuff, two more questions then:
[I have increased points to 400 too]

01.
Do you guys know of a good spider/robot that I could use to test the site [I need to make sure that particular areas are definatly found and others not found at all]

02.
David, you mentioned "creating a list of URLs to refer to these records in your database", would this mean just making a page that is filled with links like:
www.mysite.com/product.php?ProductId=123
www.mysite.com/product.php?ProductId=124
www.mysite.com/product.php?ProductId=125
www.mysite.com/product.php?ProductId=126
www.mysite.com/product.php?ProductId=127
if that was the case, how would you get the spider/robot to look at that page, without it being a hard link?

Thanks again guys !
0
 
LVL 3

Expert Comment

by:passmark
ID: 11780952

1/
My company sells the Zoom Search Engine that you could use to spider you site. But it is only free for small sites.
See, http://www.wrensoft.com and http://www.wrensoft.com/zoom/screen1.html
You just point it at your home page and hit the start button.

Otherwise there are some free alternatives on the web which may not be as easy to use (in my rather bias opinion).

2/
Yes. But here is your real problem.
Just because Google *could* index your site, doesn't mean it will. You will have the same problem as many other web designers. Namely, how do you optimize your site to get good placement (or even any placement at all) in Google? Google might also penalise or ignore pages that look like they have no significant content. So I don't think a straight list of links is going to look very interesting for Google. Whole books (and websites) have been written about placement optimization. Do a search for "Search Engine Optimization".

David
0
 
LVL 24

Expert Comment

by:duz
ID: 11781421
d2kagw -

>Do a search for "Search Engine Optimization".

Don't do that because 99% of what you will read will be complete rubbish or out of date information. Search the "Online Marketing" channel here on Experts Exchange for more reliable and up to date answers.

- duz
0

Featured Post

Are You Using the Best Web Development Editor?

The worlds of web hosting and web development are constantly evolving. Every year we see design trends change, coding standards adapt and new frameworks/CMS created. With such a quick pace of change it’s easy to get lost trying to keep up.

See if your editor made the list.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
Some of the SEO trends we might expect in 2017.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

615 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question