Implementing a java database work queue
Posted on 2007-11-16
We currently have two systems. The first system is responsible for processing incoming data. When it's processing is complete, the result is a record in a database table . The second system is responsible for picking up the completed work and forwarding the data to it's recipient (via a number of different methods). The record is then removed from the queue table and placed into a log. Currently the systems communicate via a JMS queue. When the first system is done with it's work, it sends a message to a JMS queue on the second server. I'd like to remove this dependancy. There are often times when the second system is down and thus never receives the JMS message or when we don't want the message processed immediately but at a later date and time. We also experience some reliability problems when the volume of incoming data is exceptionally high. I'd want to implement a process on the second system that reads the database table for records that are marked as ready to process. There will be several hundred of these a minute. I'm at a complete loss as to how to implement this. Can anyone point me to a design pattern or example? I obviously need to process a fairly large number of transactions in a timely manner. The time to process each transaction varies greatly depending on who the data is being sent to and how they are receiving it. Some transactions complete in microseconds, while others can take several minutes. I also need to ensure that each database record is only processed once.
Also, we are using Oracle and BEA Weblogic.