handling/parsing LARGE text files in c#


What is the best way to handle/parse large text files in c# (i.e. 500+ MB). Here's what the process would look like; there are large text files comming in, and going out, 24 hours a day 7 days a week. These files (i.e. EDI files, format is X12) have to be parsed properly and passed on to next process. File sizes vary but they are large. So, I was thinking, which way should I go...

I thought using unsafe technique in c#, also thought about using asyncronious StremReader/StreamWriter, thought about using both of these, and I still cannot deside. Any advise is highly appreciated.

LVL 13
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

davidlars99Author Commented:
ohh almost forgot... for parsing and enparsing, there will be a custom "Regex" engine developed.

davidlars99Author Commented:
article on this website says that changing strings using unsafe code is a bad aidea


In the .NET Framework, strings are immutable, meaning that once created, they cannot be changed (some counter this assertion stating that strings can be changed using unsafe code and direct memory access, but doing so is a very bad idea). As such, instead of a String, the first parameter in the P/Invoke declaration is a mutable StringBuilder, marshaled as an unmanaged LPWSTR. You also need to ensure that StringBuilder is large enough to hold the path of the folder, which on a Win32 system is at most 260 characters. The resulting C# method looks like the following:
can open file in FileStream and use binary mode - dynamic direct access, good for large file.

you have to workout many functions related to bytes reading and translating, as well as encoding type (System.Text.Encoding).


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Joni KettunenSenior Software EngineerCommented:
davidlars99Author Commented:
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.