I'm using a streamreader to parse through RADIUS logs and extract info for billing. The log's stop events come in various forms and I've managed to work out the logic to grab what I need, no problem. When I run it on a small file (1 Mb) it runs very fast, processing over 25 events per second. However when I run it on a months worth of data (30 Mb), it is extremely slow, processing 3 events per second. This leads me to believe it is not a bottleneck in my logic to process the files, but rather a problem with StreamReader on large files.
I don't think it can be threaded because I need to move line by line, grabbing what I need. Would it be better to use FileStream, which I've researched and supposedly runs slower than streamreader. Or is there any way to consistently break up a large file into smaller pieces for processing. Or am I completely wrong and there is still a better way to do this.
(Coming soon, www.ITBrainPower.com
, earn computer gear for answering questions)