handling/parsing LARGE text files in c#

Posted on 2006-03-26
Medium Priority
Last Modified: 2010-05-19

What is the best way to handle/parse large text files in c# (i.e. 500+ MB). Here's what the process would look like; there are large text files comming in, and going out, 24 hours a day 7 days a week. These files (i.e. EDI files, format is X12) have to be parsed properly and passed on to next process. File sizes vary but they are large. So, I was thinking, which way should I go...

I thought using unsafe technique in c#, also thought about using asyncronious StremReader/StreamWriter, thought about using both of these, and I still cannot deside. Any advise is highly appreciated.

Question by:davidlars99
  • 3
LVL 13

Author Comment

ID: 16295320
ohh almost forgot... for parsing and enparsing, there will be a custom "Regex" engine developed.

LVL 13

Author Comment

ID: 16295363
article on this website says that changing strings using unsafe code is a bad aidea


In the .NET Framework, strings are immutable, meaning that once created, they cannot be changed (some counter this assertion stating that strings can be changed using unsafe code and direct memory access, but doing so is a very bad idea). As such, instead of a String, the first parameter in the P/Invoke declaration is a mutable StringBuilder, marshaled as an unmanaged LPWSTR. You also need to ensure that StringBuilder is large enough to hold the path of the folder, which on a Win32 system is at most 260 characters. The resulting C# method looks like the following:

Accepted Solution

Kelmen earned 1000 total points
ID: 16297799
can open file in FileStream and use binary mode - dynamic direct access, good for large file.

you have to workout many functions related to bytes reading and translating, as well as encoding type (System.Text.Encoding).


Assisted Solution

by:Joni Kettunen
Joni Kettunen earned 1000 total points
ID: 16298302
LVL 13

Author Comment

ID: 16487500

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article series is supposed to shed some light on the use of IDisposable and objects that inherit from it. In essence, a more apt title for this article would be: using (IDisposable) {}. I’m just not sure how many people would ge…
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
Loops Section Overview
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question