How to speed up the creation of output records in a C# console application using C# with VS2010?

Posted on 2014-01-12
Medium Priority
Last Modified: 2014-01-16
I developed a C# Console application (see attachment) using VS2010 that reads an input file with no file extension and creates output records.
Do you know how I could improve the speed of this application since it takes almost 1 minute to write each output record.

Then the last file created (with the file extension ard.out) is written out about 30 minutes later.
Question by:zimmer9
LVL 40

Assisted Solution

by:Jacques Bourgeois (James Burger)
Jacques Bourgeois (James Burger) earned 1200 total points
ID: 39775314
Using constants for recurrent string values such as "COMMENT: CHECK NUMBER #" sometimes help. It depends on what the optimizer does when compiling.

You are using a lot of variables that you recreate each time through the loop. Declaring them before entering the loop might already help you a bit.

But the main problem is probably the amount of concatenation. Concatenations are inherently slow. The most common way of saving time in such a case is to use a System.Text.StringBuilder object to build the string. Although at first sight it might see as it takes more code to write, because the way it works the difference is always enormous:
System.Text.StringBuilder builder; // Define the object before going in the loop
builder.Append( "COMMENT: CHECK NUMBER #"); // Use your constants here if you go that route
builder.Append( CheckOrderNum);
builder.Append( "GROUP_FIELD_NAME:CheckNumber");

Open in new window

Instead of creating a lot of variables, you are creating a single object. You are also writing the whole file with only one WriteLine.
LVL 36

Assisted Solution

by:Miguel Oz
Miguel Oz earned 600 total points
ID: 39775774
Adding to James comments you need to check how you interact with the following file:
string indFilePath = System.IO.Path.Combine(MyGlobals.BASE_DIR, MyGlobals.BASE_FILE_Name + ".ard.ind");
                            using (StreamWriter sw = new StreamWriter(indFilePath, true))

Currently this file is open in the loop, you are better off opening only once at the start of the loop.

1) Keep in mind that you are using Ling to manipulate bytes and that may cause you some issues as well.
2) Use a Performance tool (VS 2010 and up Ultimate edition has one for free) to check which lines are performing badly or you can use the trial version of Red Gate performance profiler (commercial tool)
LVL 13

Accepted Solution

Naman Goel earned 200 total points
ID: 39775953
agree with James and Mas, also please don't use

parentDirectory.GetFiles() inside foreach loop

line #86

do something like this:

FileInfo[] fileInfos= parentDirectory.GetFiles();

    foreach (FileInfo file in fileInfos)
Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

LVL 40
ID: 39786975
Strange that the accepted solution is the one that got the smaller number of points.

Author Comment

ID: 39786981
my bad, can I correct the error of my ways
LVL 40
ID: 39787208
No problem.

I am here to give back, to help, not for the glory :-).

As a programmer who needs to concentrate on the détails, I just found it strange.

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

In real business world data are crucial and sometimes data are shared among different information systems. Hence, an agreeable file transfer protocol need to be established.
Hello there! As a developer I have modified and refactored the unit tests which was written by fellow developers in the past. On the course, I have gone through various misconceptions and technical challenges when it comes to implementation. I would…
In this video I will demonstrate how to set up Nine, which I now consider the best alternative email app to Touchdown.
Free Data Recovery software is an advanced solution from Kernel Tools to recover data and files such as documents, emails, database, media and pictures, etc. It supports recovery from physical & logical drive after a hard disk crash, accidental/inte…

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question