I have a program that generates data records in XML every hour. Each data record is dropped into a temporary directory as a 3KB XML file. This record is then merged to a larger record list using MSXML DOM objects. Over months of running, the XML file has hit 60MB. Even with smart pointers and careful memory cleanup, I'm finding that the memory footprint of the program is getting dangerously large. This seems unavoidable, since DOM loads the entire document into memory, and the document inevitably grows in size each time I integrate a data entry to it.
If this system is meant to run 24x7 for several years, we will almost certainly run out of memory as the XML file grows in size.
I did some research into this issue, and it looks like the SAX API *might* present an alternative. I have *never* done development with SAX, so before I take the time to write up a proposal and research the API, I'd like to know if anyone has successfully used MSXML's SAX API for merging two XML documents.
I found an example on IBM's website that shows merging XML files using a java-based SAX implementation: http://www.ibm.com/developerworks/xml/library/x-tipstx5/
. Can I achieve something similar with MSXML?