Java Design Question: Dealing with very large database table efficiently
Posted on 2008-11-10
I need to implment a new application that reads and publishes the contents of 1 very large database table on start up. The application will be a standalone app, and when it starts, it must query a database, and publish 20 million rows to a downstream application. The application I am writing is not required to cache the rows, it will just take each database record, convert it to a new format, and then publish it to the network. It needs to get through the 20 million rows as quick as possible.
I would be particularly interested in any open source solutions that may exist.
Thanks in advance,