Solved

Memory leak???? !dumpheap reports VERY strange results!

Posted on 2011-03-14
26
998 Views
Last Modified: 2013-12-17
I'm getting a massive memory leak in my app and I'm having a REALLY hard time figuring out where it's coming from.  When I use !dumpheap, and then a !gcroot command I get this:

!gcroot 1a916c1c
Note: Roots found on stacks may be false positives. Run "!help gcroot" for
more info.
ebx:Root:01a1e1f0(System.Windows.Forms.Application+ThreadContext)->
1d0b2450(System.Object[])->
1a911da0(foo.ComponentSubEditorUserControl)->
1a916c1c(foo.FooMemoryLeak)

where foo.MemoryLeak is the object that I'm using to track the memory leak.  Each time I run the action, it creates a form that includes foo.CompoonentSUbEditorUserControl.  This user control creates one instance of foo.FooMemoryLeak.  I close the form, pause my app and do a:

!dumpheap -type foo.FooMemoryLeak

and each time it reports an extra FooMemoryLeak object.  When I use the !gcroot command it shows System.Windows.Forms.Application+ThreadContext as the root that's holding onto my object.

What the heck is Application+ThreadContext and why is it hanging onto my user control (and thereby ALL of the objects that the user control references)?

REALLY FRUSTRATING!
0
Comment
Question by:kalliopi
  • 12
  • 10
  • 4
26 Comments
 
LVL 18

Expert Comment

by:deighton
ID: 35136885
This isn't really an answer ....

there are various articles on the net about storing objects in ThreadContext, I've not ever used that, but have you tried searching your code for anything like the stuff at

http://geekswithblogs.net/gavin/archive/2007/07/27/114228.aspx

for example

it sounds like someone is storing references to your control then keeping them.

 Do you have any third party controls?  

0
 
LVL 11

Expert Comment

by:azarc3
ID: 35136976
To me, System.Windows.Forms.Application+ThreadContext looks like a hook into a performance counter.

What is happening is  easily explainable; why it's happening is not, and I couldn't know that without some persistent datamining of your results.

Basically (even though you've closed your Form) the Garbage Collector hasn't yet recycled its memory allocation, nor possibly that of any objects that it instantiated. Unless you've been tweaking the memory management settings (and you really shouldn't do that) the memory pressure hasn't built up to a point yet where the Garbage Collector needs to reclaim now unused allocations. IN ADDITION TO THAT, any object that still has a valid reference pointing at it will not be collected until that reference is destroyed.

The 2 most common reasons (in my experience) why this would happen are:
1. Improperly handled unmanaged resources (think: anything that implements IDisposable() or has a .Close() or .Dispose() method)

2. Objects declared with the "Shared" keyword; these only go out of scope when the application domain is disposed.


From MSDN:
Besides what survived on the managed heap, what gets allocated in Gen 0 is also part of your process's committed memory. If Gen 0 is allowed to grow big before the next garbage collection occurs, you may also observe committed memory being high due to this issue. This happens a lot more often on 64-bit Windows than on 32-bit. The !eeheap –gc SOS command will show you how big Gen 0 is.


What If Objects Survive?
Sometimes developers believe that their objects should be dead, but the GC doesn't seem to clean them up. The most common causes for this are:
•There are still strong references to the objects.
•The objects were not dead the last time their generation was collected.
•The objects are dead, but a collection for the generations in which those objects live has not been triggered yet.

Link to article for deep details: http://msdn.microsoft.com/en-us/library/cc163528.aspx
0
 
LVL 18

Expert Comment

by:deighton
ID: 35137122
how big is the leak? you say it is massive, so the garbage collector should have run and removed un-reachable objects

 in any case you may have already tried gc.Collect to force collections, and if repeated forced collections do not free up memory, you probably do have a leak.

In my experience large data not disappearing means it is remaining referenced and surviving garbage collection, as you seem to have found.  

We had a problem and were told by 'consultants' that 'bad practices' were slowing garbage collection, which wasted a lot of time actually, so don't go down that route in a hurry.

I found the following helpful
http://blogs.msdn.com/b/tess/archive/2006/01/23/516139.aspx
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35137900
The form loads about 100 Mb of stuff.  Then the form closes and all of the references to it in our code are released...  Except for the one from the thread context - whatever the heck that is. So, it never gets collected, even with an explicit GC.Collect().  Within a few minutes we get an OutOfMemory exception.

Thanks for the help, but STILL Very Frustrated!
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35137993
That means you've still got a strong reference to something somewhere in that Form or one of the child objects it instantiated. Are you absolutely certain that EVERYTHING which implements IDisposable() in the Form's executable code is properly handled? Any object  that has a .Close() or .Dispose() method should be wrapped in one of the following blocks depending on the language:

C#:
using(var resource = new resourceType ())
{
    //Insert code to work with resource.
}


VB:
Using resource As New resourceType
    ' Insert code to work with resource.
End Using


Are you ABSOLUTELY certain that nothing instantiated by the form has a static or Shared keyword? If you find something that does then you'll want to investigate that on the child objects.
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35138151
I have explored all of those things.  I'm using the sos.dll memory management tools in VS and the only strong reference is the one reported in my original questions:

ROOT: 01a1e1f0(System.Windows.Forms.Application+ThreadContext)->
1d0b2450(System.Object[])->
1a911da0(foo.ComponentSubEditorUserControl)->
1a916c1c(foo.FooMemoryLeak)

The whole problem here is that these are not my references.  The ThreadContext is the one with the strong reference, and since I don't know what this is, or why is holding onto a strong reference to my user control, I don't know how to free it.
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35138253
Loading 100MB of stuff shouldn't cause an out of memory exception... unless you already have an additional low memory situation. It sounds like something is still growing and finally causing the exception.

Are you loading any physical file content into the user control using a buffer? That's one of the only two times I've seen something like what you're describing happen. The other time was when a colleague was using a 3rd party control that had a bad bug in it.
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35138323
I have a form.  There's a button on the form.  I click the button and it pops up a window that contains a bunch of stuff, including the UserControl in question.  The user control loads 100 MB of objects out of the database and references them directly in a property.

The form closes, and the system disposes of everything, but doesn't free the 100 MB of memory as expected.  I run the action again, and now the app is using 200 MB of memory.  Again, I close the form, and the memory doesn't get freed.  I repeat this process until I get an OutOfMemory exception.

When I use the SOS memory viewer tool in VS I get the stack above which shows that the Application+ThreadContext is holding onto a strong reference to the user control (which in turn is holding onto 100 MB of stuff).  The question I'm trying to figure out is what is this "Application+ThreadContext" and why is it holding onto this one random user control.

As far as I can tell, there's nothing special about it.  It's one of 5 user controls being "saved".  There are no static (shared) references.  It makes no sense to me.
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35138373
Are you using straight ADO.NET or an ORM?
0
 
LVL 18

Expert Comment

by:deighton
ID: 35138499
are you able to see the code for the user controls?  Could it be storing a reference somewhere?  

If you go backwards and forwards and the control is never released, then it sounds like a problem in the control.  

Does this control always produce a problem on any form?  

0
 
LVL 6

Author Comment

by:kalliopi
ID: 35138528
Yes, it's all my code.

Yes -but where?  Usually, when something is storing a reference, SOS reports it, and it's easy to track down.  In this case, the only reference is from this Application+ThreadContext, and I have no idea what/where/why that is.  That's the whole problem.  I'm not sure about the problem on any form - I'm still playing with it...  I'll update as I find more on this end.
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35138544
It's an ORM.  But I should point out that this application is a well tested, well used app, with almost 500K lines of code - and this is a new memory leak (just appeared in the last few weeks).
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35138594
Again, are you using ADO.NET or an ORM?

I ask because we've seen some situations where tools like NHibernate may be not release a Unit Of Work properly.
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 6

Author Comment

by:kalliopi
ID: 35138605
Again - It's an ORM. XPO from Developer Express.
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35139128
Sorry! I must have been reading/posting when you answered the first time.

I just realized something... are you spawning any child threads or using asynchronous calls? It could be that one of those calls isn't finishing before the form is closed... I'm not sure whether or not closing the caller before the callee completes would result in the behavior you're seeing.
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35139397
No worries.  

I was actually doing some work with BackgroundWorker, but my analysis showed me that the threads were finishing, and even disabling that code didn't solve the problem.

I've been shooting in the dark like this for 3 days now, and what's frustrating is that the answer is right in front of me, if I can just figure out what the heck this "ThreadContext" is, and I can't find any documentation on it - or how/why it would be holding onto my usercontrol.
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35139424
"ThreadContext" sounds like an instrumentation counter to me... are you using the Visual Studio Profiler or some other such tool that monkeys with your symbols or might inject its own instrumentation? If so, what happens when you debug with those disabled?
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35139596
I'm not doing anything like that.  To me, all the code looks relatively ordinary (very similar to lots of other code that has no trouble).
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35139705
That first blog entry that I posted suggested
1. Getting back to where you find the lingering strong reference
2. Running !finalizequeue.

If the problem is that the finalizer queue is not able to finish then this should reveal the problem.

http://msdn.microsoft.com/en-us/magazine/cc163528.aspx look for the section starting from What If Objects Survive?.
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35139839
Found another one that might help... warning -- this is a very long and involved post: http://blogs.msdn.com/b/tess/archive/2006/01/23/516139.aspx .  At best this is a "one-off" post as it's talking about Web Pages and not WinForms... but it presents a very good exercise/example about tromping through heaps (like you're doing), and near the bottom is where it gets interesting...

"If you are interested, here is the code that caused the problem

public class WebForm1 : System.Web.UI.Page
{
      public static MyClassThatHasEvents MyStaticObject = new MyClassThatHasEvents();
      
      private void InitializeComponent()
      {    
            this.Load += new System.EventHandler(this.Page_Load);
            MyStaticObject.StuffHappened += new StuffHappenedEventHandler(this.MyStaticObject_StuffHappened);  // <-- [azarc3: this line here is the problem; there's no corresponding "-="]
      }
}"


Any situations like that on the User Control or Form that hosts the User Control?
0
 
LVL 11

Expert Comment

by:azarc3
ID: 35139962
Another good one that reveals where ThreadContext comes from (at least in one scenario): http://msmvps.com/blogs/senthil/archive/2008/05/29/the-case-of-the-leaking-thread-handles.aspx

Again, the good part is at the bottom of the initial post:

" !CLRStack
OS Thread Id: 0x2c90 (3)
ESP       EIP
0106ea74 7b06b934 System.Windows.Forms.Application+ThreadContext..ctor()
0106ea78 7b06b8cc System.Windows.Forms.Application+ThreadContext.FromCurrent()
0106ea80 7b06b885 System.Windows.Forms.WindowsFormsSynchronizationContext..ctor()
0106ea90 7b06b7b2 System.Windows.Forms.WindowsFormsSynchronizationContext.InstallIfNeeded()
0106eabc 7b06a09b System.Windows.Forms.Control..ctor(Boolean)
0106eb30 7b068f75 System.Windows.Forms.Label..ctor()
0106eb3c 00d5012b LanguageFeatures.Program.<Main>b__0()
0106eb44 793b0d1f System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0106eb4c 79373ecd System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0106eb64 793b0c68 System.Threading.ThreadHelper.ThreadStart()
0106ed8c 79e7c74b [GCFrame: 0106ed8c]

As the stack dump shows, the ThreadContext was created as part of System.Windows.Forms.Control's constructor to set the synchronization context of the thread to WindowsForms. The problem was that created context didn't die when the thread died. Some more poking around with Reflector showed that the instance is removed from static Hashtable when the thread receives a quit message (via Application.ExitThread, for example). The threads in the app were not pumping messages, so the ThreadContexts kept accumulating in the Hashtable, keeping the associated Thread handle open. Here's some code that demonstrates the problem.

class Program
{
   static void Main(string[] args)
   {
      while (true)
      {
         new Thread(delegate() { new System.Windows.Forms.Label(); }).Start();
         Thread.Sleep(100);
      }
   }
}
 

[azarc3 emphasis] Bottomline - don't create controls from non-message pumping threads, especially if you create lots of threads over the lifetime of the application.
It seems obvious in retrospect, but in the application's case, it was not using the control visually, so it didn't really need WM_PAINT or the hundred other messages that a control needs to work. The fix was to move control creation to a GUI (message pumping) thread and then Invoke/BeginInvoke from the other threads."
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35142308
So - it turns out that the problem is that the base class of the user control made a call to:

Application.AddMessageFilter(this)

in it's constructor.  This causes the Application context to hold onto a strong reference to "this".  Therefore, none of the normal disposal actions ever get called, and the control is never freed.

How the heck are you supposed to unwire that call (with Application.RemoveMessageFilter(this) if there is no deconstruction, or disposal???

I should probably delete this question and pose the new question in a new thread, but I'll leave it here for now in case anyone has a great ideas on this.
0
 
LVL 6

Accepted Solution

by:
kalliopi earned 0 total points
ID: 35142462
Here's the code that fixed the problem:

 
// Original Code:


        public CustomBaseUserControl()
        {
            Application.AddMessageFilter(this);
        }

// New Code

        public CustomBaseUserControl()
        {
        }

        protected override void OnHandleCreated(EventArgs e)
        {
            base.OnHandleCreated(e);
            Application.AddMessageFilter(this);
        }

        protected override void OnHandleDestroyed(EventArgs e)
        {
            base.OnHandleDestroyed(e);
            Application.RemoveMessageFilter(this);
        }

Presto Chango - problem solved.  Thanks for all the help.

Open in new window

0
 
LVL 18

Expert Comment

by:deighton
ID: 35146213
that's interesting, thanks for letting us know how you solved it.

We had a problem like this and for some unknown reason it was intermittent, and we couldn't reproduce it on site.  It led to consultants recommending a complete code re-architecture, which was a bit stressful to be honest.  Luckily a later stress test repeated the problem and it was traced to a third party control.  The control vendor swears blind that it is down to the garbage collector, his later version seems to fix the problem though.  

I got to know a bit about the garbage collector though!
0
 
LVL 6

Author Comment

by:kalliopi
ID: 35186631
np.  Very odd/unexpected.
0
 
LVL 6

Author Closing Comment

by:kalliopi
ID: 35221369
My answer solved the problem.  The other's did not.
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

In my previous article (http://www.experts-exchange.com/Programming/Languages/.NET/.NET_Framework_3.x/A_4362-Serialization-in-NET-1.html) we saw the basics of serialization and how types/objects can be serialized to Binary format. In this blog we wi…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now