Code Preformance Issue

OK guys, sorry for the long post but there lots here! Thanks in advance!!!

Server Info:
Java 1.5
Macromedia JRUN
MySQL 5

Data Structure

TABLE: "cpList"
All employee competiencies with a parentID column to link competiencies to their parent competencies, this allows for unlimited levels of competiencies.

TABLE: "cpPosition"
All employee positions

TABLE: "cpCompetiencyToPositionMatch"
Matches competiencies to employee positions

TABLE: cpCompetencyToEmployeeMatch
Matches specific competiencies to specific employees as well as contains the date the employee achieved this competiency.

TABLE: cpCompetencyToTopicMatch
Matches competiencies to training topics


Now, there are many different places around the site where I need to display competiencies and there parent folders each, sorted, filtered, and displayed in a different way.

So here is how I am doing it now:

CompNode Class:
  1. Loads all of the current competency's child competencies.
  2. Holds a reference to this competency's parent.
  3. Holds the title of the competency.

Abstract CompNodeFilter Class:
  1. Contains a filter(CompNode cn) method that retruns true or false if the node should be filtered out of the current search.

Abstract CompNodeRenderer Class:
  1. Contains a render(ConpNode cn) that returns the HTML to be displayed for this node.


So what I am doing is loading all the "root" competencies (in other words, those with a parentID of 0) and then the CompNode class handles the recurrsive loading of all the child competencies. Then depending on the page, I apply various Filters(CompNodeFilter) and then apply a specific Renderer(CompNodeRenderer) to display everything that was loaded. For each competency I also check if the current user has completed the competency as well as when they completed it. I also find all the topics assioated with the competency and render a javascript dropdown with links to the topics.

The problem now is that everything is SLOOOOOWWW.. I have about 5000 records in the cpList table and I am usually displaying 500 - 2000 records on each page. After all said and done any page that loads competencies takes about 20 - 45 seconds to load and the project managers are hounding me to make it load faster! When I check the page properties the actuall pages are about 300KB so I figured 15 - 45 seconds was not to bad!!

Does anyone have any suggestions of how I may be able to speed things up or different ways I can go about loading and displaying everything.

Here is some testing I have done that proves our site is pretty fast, but I need to make it faster :(


Using Firefox, with cache disabled

SITE               SIZE(KB)        TIME              B/ms              
Google            17                 1.14s              14.9
Yahoo             40                 4.41s                9.0
Basecamp       21                 4.23s                5.0
Canada.com    59                 5.11s              11.6
Canoe.ca        20                 4.46s                4.5
Rogers.ca       26                 4.72s                5.5

OUR SITE       300                19.63s            15.3   << Look! Faster than EVERYONE!