Advertisement

09.06.2008 at 01:34PM PDT, ID: 23709179
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

9.2

Processing millions of files+data base inserts

Asked by topdog770 in .NET Framework 2.x, C# Programming Language, Oracle 10.x

Tags: , ,

Hi!
I'm looking for pattern(s), sample code or ideas on how to process millions of flat files as quickly as possible.

The process flow is:

1) Iterate through a directory structure containing close to 3 million files in numerous sub folders
2) for each xml doc ( one xml doc per folder )
3)  insert data base information
4)  foreach tif file in folder
5)  insert data base information
6)  combine current tif file into one combined tif file for each xml file
7) if any part of steps 2 through 6 fail, backout all operations for the current xml doc

I'm currently using the thread pool to spawn off a thread for each step 2..

Any help or ideas would be great!  I'm currently processing about 20,000 files a minute but really need to do more.

I haven't mentioned environment and let's assume xp desktop machine with dual processors, with Oracle database.  
(server hardware hasn't been identified yet.)

Thanks!Start Free Trial
[+][-]09.06.2008 at 01:56PM PDT, ID: 22408962

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]09.08.2008 at 09:35AM PDT, ID: 22419070

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]09.08.2008 at 04:33PM PDT, ID: 22422555

View this solution now by starting your 7-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

 

About this solution

Zones: .NET Framework 2.x, C# Programming Language, Oracle 10.x
Tags: C#, threading, patterns
Sign Up Now!
Solution Provided By: svdl
Participating Experts: 2
Solution Grade: A
 
 
 
Loading Advertisement...
20080716-EE-VQP-32 - Hierarchy / EE_QW_EXPERT_20070906