First instrument your program so it counts the number of bytes of log file read in, the elapsed time, and prints out the megabytes/second processed. You can't improve what you don't measure.
Then run the program as is, then with the symbol lookup feature commented out. That will reveal if the problem is in there or somewhere else.
If that *is* a major bottleneck, switch to some faster symbol lookup method. I didnt look too closely at your code, but at the very least you should keep the symbols in memory. Better yet, sorted, so you can do a binary search instead of a linear search.
The overall afterward system slowdown is probably due to the disk cache getting flushed by all the I/O activity. There isn't much you can do about this short of getting more RAM.. You could have your program do cacheless disk i/o, but that would slow down your program even more.
Main Topics
Browse All Topics
by: jaime_olivaresPosted on 2007-11-29 at 21:31:38ID: 20380078
well, you can't have a quick response with this huge code, maybe you can focus on the file-reading code and the processing portion.
A tipical bottleneck in processing a file, is when you read it record-by-record. You can accelerate meaningfully by using a bigger buffer, that is, reading 100 by 100 records, or maybe more, you can reduce time to 1/5 or 1/10 according to my experience.