Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Right solution for persisting the huge amount of data

Posted on 2006-11-21
8
Medium Priority
?
188 Views
Last Modified: 2010-03-31
Hi Experts,

Recently I have got quite a challenging task. I did not have an experience dealing with huge amount of data before. So I'm looking for good solution for it. Basically as a source I have an xml data (200Mb file) it's approximately 50000 objects. I'm planning to use Weblogic as an application server, hibernate as a persistence layer and Oracle. I'm not sure whether hibernate is the right technology for it but I want to give it a go. If you had similar experience before I will appreciate if you can share it with me.

Thank you in advance
0
Comment
Question by:worldserga
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
8 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 17989963
If it's xml then it's tagged text and should compress well
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 17989986
You are streaming the data and using hibernate, hibernate probably would not create overhead:

Use SAX parser to parse the elements and create as soon as you find complete data and call hibernate to persist that object
0
 
LVL 92

Expert Comment

by:objects
ID: 17990315
this covers what u need to store the xml directly

http://www.hibernate.org/73.html

or use Ajay-Singh's suggestion and parse the xml and use hibernate to persist the individual objects
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 1

Author Comment

by:worldserga
ID: 17997770
Basically I parse an xml data using XMLBeans. After the parsing there are 50000 objects that I need to persist. The only thing I'm concerned about is performance. Do you guys tried to persist 50000 objects?? Is there any performance issues??
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 18001442
> Do you guys tried to persist 50000 objects?? Is there any performance issues??
No
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 18001446
if you are using hibernate keep committing the transaction
0
 
LVL 1

Author Comment

by:worldserga
ID: 18004140
Ajay-Singh I appreciate your help. And I will accept your answer. Do you have any figures how long does it take to persist 50000 objects??
0
 
LVL 23

Accepted Solution

by:
Ajay-Singh earned 1005 total points
ID: 18004274
> Do you have any figures how long does it take to persist 50000 objects??
It can never meet the plain old jdbc numbers, but it would be somewhere around 50% to 100% more than that.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article is the first of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article explains our test automation goals. Then rationale is given for the tools we use to a…
Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
Viewers learn about the third conditional statement “else if” and use it in an example program. Then additional information about conditional statements is provided, covering the topic thoroughly. Viewers learn about the third conditional statement …
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
Suggested Courses

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question