Solved

Multi threaded windows service - memeory leak

Posted on 2012-03-19
16
273 Views
Last Modified: 2012-06-18
Hi,

I have a vb .net multi-threaded windows service.  There are 3 main functions that carry out various work and database reading/writing and these "worker" functions loop indefinitely.

When the service starts it creates a predefined number of each thread i.e. 5 threads are created and run Function1, 5 created and run Function2 and 5 for Function3.  So, I have 15 threads altogether, each doing their job, looping and starting again to process outstanding work from the database.

In each function, I have a label at the start i.e.
StartOfFunction:

Open in new window

and at the  end of the function we have
Goto StartOfFunction

Open in new window

.  I have error handling with try catch block which seems to work pretty well as I don't get any situations where exceptions means that the thread exits the function.

The service runs ok for about 24 hours but slowly builds in memory used and requires a restart which, obviously, is not ideal. So it seems that there is a small memory leak somewhere along the way.

I'm not sure what I need to be explicitly doing in order to prevent this, is there certain data types I need to manually dispose or set =nothing? Should I call something before the loop back to the start?  

I'm using a variety of different objects, I can provide exact details if neccessary, but summary is... string, integer, dataset, datatable, XmlTextReader, WebClient, etc. etc.

The functions can also make calls to other functions to perform database operations or xml transformations etc.

Let me know if you need any more info, but any advice would be greatly appreciated.

Many thanks
0
Comment
Question by:cp30
  • 7
  • 5
  • 4
16 Comments
 
LVL 25

Expert Comment

by:SStory
ID: 37737387
Yes, if it is something that needs disposing of you should dispose of it.  There is a garbage collection.  I have also noticed that the memory in DotNET apps sometimes seems to only be reclaimed when needed so it may not be a real problem.  Also depending upon what version of Windows it may not be reporting it correctly.  Of course if you are concatenating strings

x="test" & x
in a loop, or something like that, you are creating a lot of string objects. They will get collected eventually by the garbage collector, but you can't know when.
You could avoid some of that by using a stringbuilder object.
You can also use something like String.Format("test {0}",x)

It would be hard to determine your leak with no code.
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 37740142
Especially dispose off the datatables and datasets as these can consume a lot of memory depending on the number of rows.
0
 

Author Comment

by:cp30
ID: 37742786
Hi,

Thanks for the tips.  I know it's difficult without seeing the actual code but it's quite involved, so I hoping to just pick up some best practices and a better understanding of where the problem may lie, given the endless loop nature of the functions.

We're running on Windows Server 2008.

So, to give an example, can anyone make comments on this code example, or is this the kind of thing I should be aiming for......

Public Function exampleFunction()

startOfFunction:


        If dac.core_dataToProcess > 1 Then
            'There is no data to process
            Thread.Sleep(5000) 'sleep for 5 seconds and start again
            GoTo startOfFunction
        End If

        'We continue to process the data
        Dim ds As New DataSet
        Try

            Dim sExample1 as String
            Dim iCount1 as Integer

            'Carry out some processing with data here and write to database

        Catch ex As Exception
            dac.core_addLog(ex.Message) 'add a log
            GoTo startOfFunction 'start again
        Finally
            ds.Dispose()

        End Try

        Dim ds1 As New DataSet
        Try

            Dim sExample2 as String
            Dim iCount2 as Integer


            'Carry out some processing with data here and write to database

        Catch ex As Exception
            dac.core_addLog(ex.Message) 'add a log
            GoTo startOfFunction 'start again
        Finally
            ds1.Dispose()
        End Try

        'We loop round and start again
        GoTo startOfFunction


    End Function

Open in new window


Also, do I need to worry about disposing objects that are in sub routines or functions that are called from the main function, or will these be handled by garbage collection as they fall out of scope when focus switches back to main function?

Thanks
0
 
LVL 25

Expert Comment

by:SStory
ID: 37743359
Personally I don't like the GOTO. It would be better to do

dim Done as boolean=false

While not done
'put code here
'no spaghetti code; a simple loop until done=true

End While

Also I don't see you doing anything with this dataset.  Just because you call Dispose() doesn't necessarily mean the memory will decrease. It just means the garbage collector will do so at a indeterminable time.  As an aside the use of case in a Java fashion is not usually done in VB. Most people use CamelCase.  Capitalize the first letter of all words and not the others.

I don't see anything else that grabs me, of course this is an example and not the real code.
0
 
LVL 25

Expert Comment

by:SStory
ID: 37743372
I also wonder why instead of:
   If dac.core_dataToProcess > 1 Then
            'There is no data to process
            Thread.Sleep(5000) 'sleep for 5 seconds and start again
            GoTo startOfFunction
        End If

why not have

While dac.core_dataToProcess > 1
         'no data to process
         Thread.Sleep(5000)
End While

But I wonder the thread safety involved in checking dac.core_dataToProcess as well.
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 37743976
Try setting the datasets = nothing as well hoping that this will cause the GC to collect these. Also handle the objects in other subs similarly.
0
 
LVL 25

Expert Comment

by:SStory
ID: 37744028
You can also force GC to collect garbage. I wouldn't do it every time though.
http://www.vb-helper.com/howto_net_force_gc.html

Maybe you could have a counter that gets incremented after the
Thread.Sleep(5000) and once that is equal to 36 (3minutes have passed) or whatever time you come up with, set the counter to 0 and force GC to collect(see the link) and let it wait another N minutes before doing it again.
0
 

Author Comment

by:cp30
ID: 37747364
Hi,

Thanks for all the comments.  I was hoping for some conclusive information, but everything seems to be people's views and opinions.  People saying that they don't like goto etc are not that helpful unless they can back that up with a reason while it may be affecting memory problems or if it's just personal preference.

I need to know things like, if I declare a string variable or an integer, do I need to set that nothing before I loop and use the code again to reduce memory leaks?  Should I dispose the datasets/datatables or should I set =Nothing, or both, are there any other types of objects that need special procedures to eliminate memory from never being released and leaking?

SStory: The code example I gave was off the top of my head, I know it didn't do anything with the dataset, I added a comment to indicate where I would do something in real application.

Are there any other typical reasons for memory leaking in .net, common mistakes people make? Or any methods to tackle this from the other end and monitor the service and detect where the memory leaks are occurring?

Many thanks in advance
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 25

Expert Comment

by:SStory
ID: 37747515
Well, sorry you don't like opinions.

Anyhow, I was trying to tell you how to force GC to collect which may show that memory isn't really leaking, just not getting collected as you think it should. I have noticed this in dotnet apps. They seem to be using a horrendous amount of memory when a lot of times the memory is available to be reclaimed, but only when needed.  GC's collect say, do it already ASAP.

From the code you've shown I see no reason for a memory leak. If you are accessing unsafe code, GDI objects or something like that then you will have to use unsafe methods to release the memory or it will cause leaks. Other than that, generally put a dispose in the finally part for appropriate objects and you shouldn't have leaks.

So my point was do you really have a leak or is it just lazy GC on collecting?
0
 

Author Comment

by:cp30
ID: 37747574
Well, sorry you don't like opinions.

Sorry, I didn't mean it to sound like that.  I was just trying to say that I know that people will have different opinions and if I followed them all then I'd be treading on my own toes before too long, so it's good to know what the "right" way to do things is.

You reference to forcing garbage collection is one that I may try, but I've just heard for so long that manual manual garbage collection is a bad idea but maybe this is one of those situations where .net doesn't know best and manual gc may help.  If I force gc in one thread, does that apply to all threads within the application or just that individual thread? (sorry if that's a stupid question)

Thanks
0
 
LVL 25

Expert Comment

by:SStory
ID: 37748014
No I think that is a good question. I really don't know the answer to it either. I'm not sure you should or shouldn't force it, but doing so even temporarily, if it has a different outcome might tell you that you think you have a leak and don't really.  I don't know if they've fixed the way these get reported in all OSs or not.
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 37774252
>Should I dispose the datasets/datatables or should I set =Nothing, or both

http:#a37740142
Especially dispose off the datatables and datasets as these can consume a lot of memory depending on the number of rows.

http:#a37743976
Try setting the datasets = nothing as well hoping that this will cause the GC to collect these. Also handle the objects in other subs similarly.

Did you read all the comments?
0
 

Author Comment

by:cp30
ID: 37779413
Did you read all the comments?

Hi, yes I did read all the comments but before I go through the the whole application to try to isolate areas where we can prevent potential memory leaks, I wanted to get confirmation on what I should be focussing on.  

Different people said to dispose of the dataset, and another to set = nothing, I was just wondering if anyone knew conclusively whether both were necessary or not, and, if they are, then in what order.  If it is a case that no-one knows for sure then that's fine and I'll just have to try and apply both and see if that solves the issues, but as it's a reasonably large task to go through the entire service code to identify I just want to make sure we have a plan of action based on the most accurate advice we can get.

Hope that makes  a little more  sense, it wasn't that I was not bothering to read the comments.

Cheers
0
 
LVL 83

Assisted Solution

by:CodeCruiser
CodeCruiser earned 250 total points
ID: 37779574
Its alright. Calling the dispose method asks the dataset or any other object to release its own memory. Setting its reference to null would mean that when garbage collector comes around, it would notice that the dataset (which may have been dispose) is not being used anywhere and will remove it from memory if anything of it is still left.

With regards to sequence, call the dispose first and then set to nothing/null.
0
 
LVL 25

Accepted Solution

by:
SStory earned 250 total points
ID: 37781553
0
 

Author Comment

by:cp30
ID: 37794937
Well you could try one of these:

http://memprofiler.com/

http://www.codeproject.com/Articles/19490/Memory-Leak-Detection-in-NET

Thanks, will try those also.
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Today I had a very interesting conundrum that had to get solved quickly. Needless to say, it wasn't resolved quickly because when we needed it we were very rushed, but as soon as the conference call was over and I took a step back I saw the correct …
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now