angel7170
asked on
Pass multiple values or string arrays in java as a parameter
Hello,
I have a JAVA program that currently fetches ID from a oracle table and pass as a parameter to cassandra query, fetches the details and inserts into another oracle table.
The problem I have is it is taking about 1 hour to process only 1000 records. This won't work as we have about more than 10 million records.
How do I supply array of IDs as a parameter into the method "ProcessData" to process it much faster? Or any suggestion on how this code could be improved that would be great.
Thank you
ExportOnlyID_2017.java
I have a JAVA program that currently fetches ID from a oracle table and pass as a parameter to cassandra query, fetches the details and inserts into another oracle table.
The problem I have is it is taking about 1 hour to process only 1000 records. This won't work as we have about more than 10 million records.
How do I supply array of IDs as a parameter into the method "ProcessData" to process it much faster? Or any suggestion on how this code could be improved that would be great.
Thank you
ExportOnlyID_2017.java
Step 1 in speeding things up is first figuring out which part is slow.
You should add:
long startTime = System.currentTimeMillis() ;
takeAction() ; <-- the thing you want to look at
System.out.println("Step X took " + (System.currentTimeMillis( ) - startTime) + " ms") ;
with "X" replaced with some meaningful string
around each of your database/cassandra calls.
I.E. Before and after each of these:
rs = pstmt.executeQuery();
or these
ResultSet rs = session.execute(stmt);
Then when you run it and it takes an hour, you'll see which of these statements is taking up the time.
I'd bet one of them is taking 55 mins of your hour. That's the piece you need to optimize - but you need to find it first.
Doug
You should add:
long startTime = System.currentTimeMillis()
takeAction() ; <-- the thing you want to look at
System.out.println("Step X took " + (System.currentTimeMillis(
with "X" replaced with some meaningful string
around each of your database/cassandra calls.
I.E. Before and after each of these:
rs = pstmt.executeQuery();
or these
ResultSet rs = session.execute(stmt);
Then when you run it and it takes an hour, you'll see which of these statements is taking up the time.
I'd bet one of them is taking 55 mins of your hour. That's the piece you need to optimize - but you need to find it first.
Doug
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
From a quick Google, it appears that you can create an ODBC connection to Cassandra.
If you can, I take a look at Oracle's Heterogeneous gateway to connect the Oracle database to the Cassandra and do everything in an "insert into select" query.
Heterogeneous gateway docs:
http://docs.oracle.com/database/121/HETER/toc.htm
If you cannot, then I think you are stuck with memory objects that hold lists of id's and abstract_html. Then you should be able to do batch processing using the lists.