Destroying XMLNode and XMLElement objects

I have a program where we create a very large (50MB+) XML file using C#.  It is taking a huge amount of memory in doing this.  It had become clear to use that the unreferenced objects were not being cleaned up and this the memory usage just kept growing.

We built a test program to verify this:

        /*----------------------------------------------------------------------
         * public void LoadDocument()
         *
         *
         * --------------------------------------------------------------------*/
        public void LoadDocument()
        {
            XmlDocument m_XmlData;
            XmlNode testnode;
            XmlElement testelement;

            m_XmlData = new XmlDocument();
            m_XmlData.LoadXml("<Test />");

            for (int i = 0; i < 2300; i++)
            {
                Trace.WriteLine("Running Loop " + i.ToString());

                testnode = m_XmlData.CreateElement("Test");
                testelement = m_XmlData.CreateElement("TestElement");
                testelement.InnerText = "Hi There";

                testnode.AppendChild(testelement);

                m_XmlData.GetElementsByTagName("Test").Item(0).AppendChild(testnode);
            }
        }

This function is called in a thread and when executed allocates about 4MB to preform.  When the function is done, and it exits the thread ends also.  However the memory allocation remains unchanged.  The memory allocated is not released.  This confirms our thought that memory was being allocated but not freed.   I know that garbage collection does not happen right away but how can I get these XMLNode and XMLElement objects to free after each interation of the loop?

When we run our larger XML file creator, it will create the 50 MB XML file but will allocate almost 1GB of RAM to do it.  It is hugely disproportionate.

We have tried having the garbage collection run at the end of each loop using GC.Collect() but it had no effect.

Any ideas out there?
LVL 2
jymmealeyAsked:
Who is Participating?
 
vo1dConnect With a Mentor Commented:
you should not create your xml structure in memory first.
write it directly into the file by using the XmlWriter.
0
 
jymmealeyAuthor Commented:
Writing it directly to the file is not an option for us at this point.  It would mean heavily re-writing and that can't happen right now.  If I can get the memory issue resolved there should be no need.

Jym
0
 
vo1dCommented:
you cant solve that memoryissue because you are bulding a tree of xmlnodes internally.
so you have dependencies between parent nodes and childnodes.
what you could try to do is splitting your structure into logical blocks and create a list of references to memorystreams.
then in your loop, create the logical block with a reference to its parent node, write that block into a memorystream and compress that stream.
after creating your data itereate through your memorystream list, decrompress each memorystreamobject and write it to a file.
thats only an appproach.
0
 
jymmealeyAuthor Commented:
I think vo1d is correct although I have not been able to fully explore all options.  As created elements are added to the document it seems to create some kind of lasting reference, although the object is no longer directly accessed by the document.  I have tested several times by chaning the element object after it is added and it make no effect on the document so it seems silly that the reference holds, but none the less it does.

I am using the XMLWriter, I had to alter the program a bit but I think it will work fine.

Thanks
Jym
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.