• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 830
  • Last Modified:

MSXML DOM Leaks Memory. Or am I doing anything wrong??

I have large XML Files that I parse and create records in a SQL Database. Each of these records are stored in the file as an item (<Item> blah blah </Item>). So, I loop thru the file reading each record. Something like this...
 Set NodeList = xmlDoc.documentElement.selectNodes("Item")
 For Each Node in NodeList
        processNode Node
        Set Node = Nothing
 Next Node
 Set NodeList = Nothing
 Set xmlDoc = Nothing
-------
The problem is, the operation is consuming a lot of memory and is not releasing, even though I am setting every object I am using to Nothing. Eventually, I run out of memory before I am done processing my file and it crashes saying "out of memory"

What am I doing wrong? Does MSXML leak memory or is it something that I am not accounting for?? Can somebody help?
0
FastTrack
Asked:
FastTrack
  • 2
  • 2
  • 2
  • +1
1 Solution
 
xsl-lsxCommented:
It's not XSL, what does your SQL code look like?
Are you using same diligent in setting everything to null?
0
 
FastTrackAuthor Commented:
Actually, I am not even sending anything to SQL at this point. As a process of isolation, I just took the XML Code, put it in a saperate program and ran it. All this program does is to open the XML File (xmlDoc.Load) and traverse thru every node in the XML File. For example, if the XML File is 1.2 MB, xmlDoc.Load locks up about 10 MB of RAM, then for every item it moves thru, the memory usage keeps increasing. By the time the program is done with the file, the process is holding about 45 MB of memory. So far, there is no database interaction, not even storing the data from the file in memory. Just a traverse thru the file, moving from one node to another.
0
 
xsl-lsxCommented:
That is normal expansion of XML-DOM.  Every part of the tree is a separate tree-node.
Then all the new stuff your adding into each item becomes a new and different tree of nodes. One-way to save some space, is to use <xsl:copy> and copy-of to reuse a tree-node. If the final output is about 7meg then the 45 Meg is about right.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
sparkplugCommented:
Hi,

You could try changing the code the following to see if it makes a difference:

Set NodeList = xmlDoc.documentElement.selectNodes("Item")
For i = 0 To NodeList.length -1
       Set Node = NodeList.Item(i)
       processNode Node
       Set Node = Nothing
Next Node
Set NodeList = Nothing
Set xmlDoc = Nothing

I know that For Each is less efficient than a normal For loop, but it shouldn't cause memory leaks. What are you doing within the processNode rountine?

>S'Plug<
0
 
BigRatCommented:
Technically there is no difference between FOR EACH (uses IEnumVariant) and item(i) since the former contains the latter. Admittedly the enumeration object could get left lying around but I don't think so.

I'd also like to know what processNode does.

And I'd like to know which DOM was created. Ie: what have you for a name in the CreateObject() for the xmlDom?
0
 
BigRatCommented:
Technically there is no difference between FOR EACH (uses IEnumVariant) and item(i) since the former contains the latter. Admittedly the enumeration object could get left lying around but I don't think so.

I'd also like to know what processNode does.

And I'd like to know which DOM was created. Ie: what have you for a name in the CreateObject() for the xmlDom?
0
 
FastTrackAuthor Commented:
The comment from sparkplug seems to fix the issue. Thanks spark. It came to my attention that the For Each Loop has an inherrent memory leak. I replaced it with the counters approach and it seems to be working, for now. I need to run further tests to figure out if everything is running as expected.

0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 2
  • 2
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now