Solved

Right solution for persisting the huge amount of data

Posted on 2006-11-21
8
182 Views
Last Modified: 2010-03-31
Hi Experts,

Recently I have got quite a challenging task. I did not have an experience dealing with huge amount of data before. So I'm looking for good solution for it. Basically as a source I have an xml data (200Mb file) it's approximately 50000 objects. I'm planning to use Weblogic as an application server, hibernate as a persistence layer and Oracle. I'm not sure whether hibernate is the right technology for it but I want to give it a go. If you had similar experience before I will appreciate if you can share it with me.

Thank you in advance
0
Comment
Question by:worldserga
8 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 17989963
If it's xml then it's tagged text and should compress well
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 17989986
You are streaming the data and using hibernate, hibernate probably would not create overhead:

Use SAX parser to parse the elements and create as soon as you find complete data and call hibernate to persist that object
0
 
LVL 92

Expert Comment

by:objects
ID: 17990315
this covers what u need to store the xml directly

http://www.hibernate.org/73.html

or use Ajay-Singh's suggestion and parse the xml and use hibernate to persist the individual objects
0
 
LVL 1

Author Comment

by:worldserga
ID: 17997770
Basically I parse an xml data using XMLBeans. After the parsing there are 50000 objects that I need to persist. The only thing I'm concerned about is performance. Do you guys tried to persist 50000 objects?? Is there any performance issues??
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 18001442
> Do you guys tried to persist 50000 objects?? Is there any performance issues??
No
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 18001446
if you are using hibernate keep committing the transaction
0
 
LVL 1

Author Comment

by:worldserga
ID: 18004140
Ajay-Singh I appreciate your help. And I will accept your answer. Do you have any figures how long does it take to persist 50000 objects??
0
 
LVL 23

Accepted Solution

by:
Ajay-Singh earned 335 total points
ID: 18004274
> Do you have any figures how long does it take to persist 50000 objects??
It can never meet the plain old jdbc numbers, but it would be somewhere around 50% to 100% more than that.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
groupSum5 challenge 5 68
wordappend challenge 8 143
Systems talking to each other 5 123
JList custom Cell Renderer refresh 15 53
For customizing the look of your lightweight component and making it look lucid like it was made of glass. Or: how to make your component more Apple-ish ;) This tip assumes your component to be of rectangular shape and completely opaque. (COD…
Java had always been an easily readable and understandable language.  Some relatively recent changes in the language seem to be changing this pretty fast, and anyone that had not seen any Java code for the last 5 years will possibly have issues unde…
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:
Viewers will learn about basic arrays, how to declare them, and how to use them. Introduction and definition: Declare an array and cover the syntax of declaring them: Initialize every index in the created array: Example/Features of a basic arr…

929 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now