java performance loading huge data
Posted on 2013-06-03
Currently in our application we have a remote database and our application database
The remote database has 800,000 rows and gets updated with more data every now and then.
When our java process runs we need to copy the data from the remote database
into our applicaiton database.
Since our java process runs 10 times a day we are seeing a performance issue.
Each time our java process runs we delete the data (800,000 + rows) from our table and insert it again from the remote database to ensure we have current data.
This is affecting our performance since it takes 20 mins to insert 800,000 rows into our table.
We want to come up with a cache to avoid this problem
maybe have a java List or a Map which contains the 800,000 + objects
and whenever the remote database has some new data, we constantly listen to it and update our List/Map.