Right solution for persisting the huge amount of data

Posted on 2006-11-21
Last Modified: 2010-03-31
Hi Experts,

Recently I have got quite a challenging task. I did not have an experience dealing with huge amount of data before. So I'm looking for good solution for it. Basically as a source I have an xml data (200Mb file) it's approximately 50000 objects. I'm planning to use Weblogic as an application server, hibernate as a persistence layer and Oracle. I'm not sure whether hibernate is the right technology for it but I want to give it a go. If you had similar experience before I will appreciate if you can share it with me.

Thank you in advance
Question by:worldserga
LVL 86

Expert Comment

ID: 17989963
If it's xml then it's tagged text and should compress well
LVL 23

Expert Comment

ID: 17989986
You are streaming the data and using hibernate, hibernate probably would not create overhead:

Use SAX parser to parse the elements and create as soon as you find complete data and call hibernate to persist that object
LVL 92

Expert Comment

ID: 17990315
this covers what u need to store the xml directly

or use Ajay-Singh's suggestion and parse the xml and use hibernate to persist the individual objects

Author Comment

ID: 17997770
Basically I parse an xml data using XMLBeans. After the parsing there are 50000 objects that I need to persist. The only thing I'm concerned about is performance. Do you guys tried to persist 50000 objects?? Is there any performance issues??
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

LVL 23

Expert Comment

ID: 18001442
> Do you guys tried to persist 50000 objects?? Is there any performance issues??
LVL 23

Expert Comment

ID: 18001446
if you are using hibernate keep committing the transaction

Author Comment

ID: 18004140
Ajay-Singh I appreciate your help. And I will accept your answer. Do you have any figures how long does it take to persist 50000 objects??
LVL 23

Accepted Solution

Ajay-Singh earned 335 total points
ID: 18004274
> Do you have any figures how long does it take to persist 50000 objects??
It can never meet the plain old jdbc numbers, but it would be somewhere around 50% to 100% more than that.

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
scoreUp challenge 14 48
strCopies  challenge 17 73
ArrayLists in ArrayLists/HashMaps? 4 39
Strange loading of website behaviour 3 23
An old method to applying the Singleton pattern in your Java code is to check if a static instance, defined in the same class that needs to be instantiated once and only once, is null and then create a new instance; otherwise, the pre-existing insta…
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now