We help IT Professionals succeed at work.

handling/parsing LARGE text files in c#

davidlars99
davidlars99 asked
on
Medium Priority
422 Views
Last Modified: 2010-05-19
Hi,

What is the best way to handle/parse large text files in c# (i.e. 500+ MB). Here's what the process would look like; there are large text files comming in, and going out, 24 hours a day 7 days a week. These files (i.e. EDI files, format is X12) have to be parsed properly and passed on to next process. File sizes vary but they are large. So, I was thinking, which way should I go...

I thought using unsafe technique in c#, also thought about using asyncronious StremReader/StreamWriter, thought about using both of these, and I still cannot deside. Any advise is highly appreciated.

Thanks!
Comment
Watch Question

Author

Commented:
ohh almost forgot... for parsing and enparsing, there will be a custom "Regex" engine developed.

Author

Commented:
article on this website says that changing strings using unsafe code is a bad aidea

http://msdn.microsoft.com/msdnmag/issues/04/04/NETMatters/

[article]
In the .NET Framework, strings are immutable, meaning that once created, they cannot be changed (some counter this assertion stating that strings can be changed using unsafe code and direct memory access, but doing so is a very bad idea). As such, instead of a String, the first parameter in the P/Invoke declaration is a mutable StringBuilder, marshaled as an unmanaged LPWSTR. You also need to ensure that StringBuilder is large enough to hold the path of the folder, which on a Win32 system is at most 260 characters. The resulting C# method looks like the following:
Commented:
can open file in FileStream and use binary mode - dynamic direct access, good for large file.

you have to workout many functions related to bytes reading and translating, as well as encoding type (System.Text.Encoding).

hth

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts
Joni KettunenSenior Software Engineer
Commented:

Author

Commented:
thanks!
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.