?
Solved

MSXML DOM Leaks Memory. Or am I doing anything wrong??

Posted on 2003-03-27
9
Medium Priority
?
815 Views
Last Modified: 2011-09-20
I have large XML Files that I parse and create records in a SQL Database. Each of these records are stored in the file as an item (<Item> blah blah </Item>). So, I loop thru the file reading each record. Something like this...
 Set NodeList = xmlDoc.documentElement.selectNodes("Item")
 For Each Node in NodeList
        processNode Node
        Set Node = Nothing
 Next Node
 Set NodeList = Nothing
 Set xmlDoc = Nothing
-------
The problem is, the operation is consuming a lot of memory and is not releasing, even though I am setting every object I am using to Nothing. Eventually, I run out of memory before I am done processing my file and it crashes saying "out of memory"

What am I doing wrong? Does MSXML leak memory or is it something that I am not accounting for?? Can somebody help?
0
Comment
Question by:FastTrack
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
  • 2
  • +1
9 Comments
 
LVL 1

Expert Comment

by:xsl-lsx
ID: 8219888
It's not XSL, what does your SQL code look like?
Are you using same diligent in setting everything to null?
0
 

Author Comment

by:FastTrack
ID: 8220439
Actually, I am not even sending anything to SQL at this point. As a process of isolation, I just took the XML Code, put it in a saperate program and ran it. All this program does is to open the XML File (xmlDoc.Load) and traverse thru every node in the XML File. For example, if the XML File is 1.2 MB, xmlDoc.Load locks up about 10 MB of RAM, then for every item it moves thru, the memory usage keeps increasing. By the time the program is done with the file, the process is holding about 45 MB of memory. So far, there is no database interaction, not even storing the data from the file in memory. Just a traverse thru the file, moving from one node to another.
0
 
LVL 1

Expert Comment

by:xsl-lsx
ID: 8220737
That is normal expansion of XML-DOM.  Every part of the tree is a separate tree-node.
Then all the new stuff your adding into each item becomes a new and different tree of nodes. One-way to save some space, is to use <xsl:copy> and copy-of to reuse a tree-node. If the final output is about 7meg then the 45 Meg is about right.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 9

Accepted Solution

by:
sparkplug earned 300 total points
ID: 8223910
Hi,

You could try changing the code the following to see if it makes a difference:

Set NodeList = xmlDoc.documentElement.selectNodes("Item")
For i = 0 To NodeList.length -1
       Set Node = NodeList.Item(i)
       processNode Node
       Set Node = Nothing
Next Node
Set NodeList = Nothing
Set xmlDoc = Nothing

I know that For Each is less efficient than a normal For loop, but it shouldn't cause memory leaks. What are you doing within the processNode rountine?

>S'Plug<
0
 
LVL 27

Expert Comment

by:BigRat
ID: 8224013
Technically there is no difference between FOR EACH (uses IEnumVariant) and item(i) since the former contains the latter. Admittedly the enumeration object could get left lying around but I don't think so.

I'd also like to know what processNode does.

And I'd like to know which DOM was created. Ie: what have you for a name in the CreateObject() for the xmlDom?
0
 
LVL 27

Expert Comment

by:BigRat
ID: 8224014
Technically there is no difference between FOR EACH (uses IEnumVariant) and item(i) since the former contains the latter. Admittedly the enumeration object could get left lying around but I don't think so.

I'd also like to know what processNode does.

And I'd like to know which DOM was created. Ie: what have you for a name in the CreateObject() for the xmlDom?
0
 

Author Comment

by:FastTrack
ID: 8226311
The comment from sparkplug seems to fix the issue. Thanks spark. It came to my attention that the For Each Loop has an inherrent memory leak. I replaced it with the counters approach and it seems to be working, for now. I need to run further tests to figure out if everything is running as expected.

0

Featured Post

Stack Overflow Podcast - Developer Story

Welcome to the Stack Overflow podcast recorded Thursday July 20 at Stack Overflow Headquearters in NYC. Your hosts today are podcast regulars Jay Hanlon, David Fullerton, and Ilana Yitzhaki, plus the quite irregular Matt Sherman (Stack Overflow Engineering Manager extraordinaire)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction In my previous article (http://www.experts-exchange.com/Microsoft/Development/MS-SQL-Server/SSIS/A_9150-Loading-XML-Using-SSIS.html) I showed you how the XML Source component can be used to load XML files into a SQL Server database, us…
I was working on a PowerPoint add-in the other day and a client asked me "can you implement a feature which processes a chart when it's pasted into a slide from another deck?". It got me wondering how to hook into built-in ribbon events in Office.
This tutorial will teach you the special effect of super speed similar to the fictional character Wally West aka "The Flash" After Shake : http://www.videocopilot.net/presets/after_shake/ All lightning effects with instructions : http://www.mediaf…
Sometimes it takes a new vantage point, apart from our everyday security practices, to truly see our Active Directory (AD) vulnerabilities. We get used to implementing the same techniques and checking the same areas for a breach. This pattern can re…
Suggested Courses
Course of the Month8 days, 22 hours left to enroll

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question