Right solution for persisting the huge amount of data

Hi Experts,

Recently I have got quite a challenging task. I did not have an experience dealing with huge amount of data before. So I'm looking for good solution for it. Basically as a source I have an xml data (200Mb file) it's approximately 50000 objects. I'm planning to use Weblogic as an application server, hibernate as a persistence layer and Oracle. I'm not sure whether hibernate is the right technology for it but I want to give it a go. If you had similar experience before I will appreciate if you can share it with me.

Thank you in advance
LVL 1
worldsergaAsked:
Who is Participating?
 
Ajay-SinghCommented:
> Do you have any figures how long does it take to persist 50000 objects??
It can never meet the plain old jdbc numbers, but it would be somewhere around 50% to 100% more than that.
0
 
CEHJCommented:
If it's xml then it's tagged text and should compress well
0
 
Ajay-SinghCommented:
You are streaming the data and using hibernate, hibernate probably would not create overhead:

Use SAX parser to parse the elements and create as soon as you find complete data and call hibernate to persist that object
0
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

 
objectsCommented:
this covers what u need to store the xml directly

http://www.hibernate.org/73.html

or use Ajay-Singh's suggestion and parse the xml and use hibernate to persist the individual objects
0
 
worldsergaAuthor Commented:
Basically I parse an xml data using XMLBeans. After the parsing there are 50000 objects that I need to persist. The only thing I'm concerned about is performance. Do you guys tried to persist 50000 objects?? Is there any performance issues??
0
 
Ajay-SinghCommented:
> Do you guys tried to persist 50000 objects?? Is there any performance issues??
No
0
 
Ajay-SinghCommented:
if you are using hibernate keep committing the transaction
0
 
worldsergaAuthor Commented:
Ajay-Singh I appreciate your help. And I will accept your answer. Do you have any figures how long does it take to persist 50000 objects??
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.