Link to home
Start Free TrialLog in
Avatar of angel7170
angel7170Flag for United States of America

asked on

Pass multiple values or string arrays in java as a parameter

Hello,

I have a JAVA program that currently fetches ID from a oracle table and pass as a parameter to cassandra query, fetches the details and inserts into another oracle table.

The problem I have is it is taking about 1 hour to process only 1000 records. This won't work as we have about more than 10 million records.

How do I supply array of IDs as a parameter into the method "ProcessData" to process it much faster? Or any suggestion on how this code could be improved that would be great.

Thank you
ExportOnlyID_2017.java
Avatar of slightwv (䄆 Netminder)
slightwv (䄆 Netminder)

I'm not a Java programmer but know enough to follow the code.  I also know nothing about Cassandra.

From a quick Google, it appears that you can create an ODBC connection to Cassandra.

If you can, I take a look at Oracle's Heterogeneous gateway to connect the Oracle database to the Cassandra and do everything in an "insert into select" query.

Heterogeneous gateway docs:
http://docs.oracle.com/database/121/HETER/toc.htm

If you cannot, then I think you are stuck with memory objects that hold lists of id's and abstract_html.  Then you should be able to do batch processing using the lists.
Step 1 in speeding things up is first figuring out which part is slow.

You should add:
                long startTime = System.currentTimeMillis() ;
                takeAction() ; <-- the thing you want to look at
                System.out.println("Step X took " + (System.currentTimeMillis() - startTime) + " ms") ;

with "X" replaced with some meaningful string
            
around each of your database/cassandra calls.

I.E. Before and after each of these:
rs = pstmt.executeQuery();  
or these
ResultSet rs = session.execute(stmt);

Then when you run it and it takes an hour, you'll see which of these statements is taking up the time.

I'd bet one of them is taking 55 mins of your hour.  That's the piece you need to optimize - but you need to find it first.

Doug
ASKER CERTIFIED SOLUTION
Avatar of Peter Kwan
Peter Kwan
Flag of Hong Kong image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial