Tips For Parsing Large Files
Posted on 2009-05-06
Often I have a need to parse and/or compare very large files (up to 1GB sometimes) and although I have plenty of experience doing so, I'd still like to know if there are any methods or objects I could use that can help get the job done faster. Usually I stick to VB languages, but am not disinclined to using PHP, Perl, Python, or command-line if need be. For VBscript, and VBA I usually stick to using multiple dictionary objects, or a dictionary of dictionaries as I've found this helps to store and compare large tables of data with a drastically reduced amount of processing time. For VB.Net I use a hashtable... Though still, sometimes these processes take hours to run. Are there any other objects I should consider using, or are these the top preformers and I'm not likely to find anything better to use?