asked on Multi threaded windows service - memeory leak
Hi,
I have a vb .net multi-threaded windows service. There are 3 main functions that carry out various work and database reading/writing and these "worker" functions loop indefinitely.
When the service starts it creates a predefined number of each thread i.e. 5 threads are created and run Function1, 5 created and run Function2 and 5 for Function3. So, I have 15 threads altogether, each doing their job, looping and starting again to process outstanding work from the database.
In each function, I have a label at the start i.e.
The service runs ok for about 24 hours but slowly builds in memory used and requires a restart which, obviously, is not ideal. So it seems that there is a small memory leak somewhere along the way.
I'm not sure what I need to be explicitly doing in order to prevent this, is there certain data types I need to manually dispose or set =nothing? Should I call something before the loop back to the start?
I'm using a variety of different objects, I can provide exact details if neccessary, but summary is... string, integer, dataset, datatable, XmlTextReader, WebClient, etc. etc.
The functions can also make calls to other functions to perform database operations or xml transformations etc.
Let me know if you need any more info, but any advice would be greatly appreciated.
Many thanks
I have a vb .net multi-threaded windows service. There are 3 main functions that carry out various work and database reading/writing and these "worker" functions loop indefinitely.
When the service starts it creates a predefined number of each thread i.e. 5 threads are created and run Function1, 5 created and run Function2 and 5 for Function3. So, I have 15 threads altogether, each doing their job, looping and starting again to process outstanding work from the database.
In each function, I have a label at the start i.e.
StartOfFunction:
Goto StartOfFunction
The service runs ok for about 24 hours but slowly builds in memory used and requires a restart which, obviously, is not ideal. So it seems that there is a small memory leak somewhere along the way.
I'm not sure what I need to be explicitly doing in order to prevent this, is there certain data types I need to manually dispose or set =nothing? Should I call something before the loop back to the start?
I'm using a variety of different objects, I can provide exact details if neccessary, but summary is... string, integer, dataset, datatable, XmlTextReader, WebClient, etc. etc.
The functions can also make calls to other functions to perform database operations or xml transformations etc.
Let me know if you need any more info, but any advice would be greatly appreciated.
Many thanks
Visual Basic.NET.NET Programming
Last Comment
cp30
Especially dispose off the datatables and datasets as these can consume a lot of memory depending on the number of rows.
ASKER
Hi,
Thanks for the tips. I know it's difficult without seeing the actual code but it's quite involved, so I hoping to just pick up some best practices and a better understanding of where the problem may lie, given the endless loop nature of the functions.
We're running on Windows Server 2008.
So, to give an example, can anyone make comments on this code example, or is this the kind of thing I should be aiming for......
Also, do I need to worry about disposing objects that are in sub routines or functions that are called from the main function, or will these be handled by garbage collection as they fall out of scope when focus switches back to main function?
Thanks
Thanks for the tips. I know it's difficult without seeing the actual code but it's quite involved, so I hoping to just pick up some best practices and a better understanding of where the problem may lie, given the endless loop nature of the functions.
We're running on Windows Server 2008.
So, to give an example, can anyone make comments on this code example, or is this the kind of thing I should be aiming for......
Public Function exampleFunction()
startOfFunction:
If dac.core_dataToProcess > 1 Then
'There is no data to process
Thread.Sleep(5000) 'sleep for 5 seconds and start again
GoTo startOfFunction
End If
'We continue to process the data
Dim ds As New DataSet
Try
Dim sExample1 as String
Dim iCount1 as Integer
'Carry out some processing with data here and write to database
Catch ex As Exception
dac.core_addLog(ex.Message) 'add a log
GoTo startOfFunction 'start again
Finally
ds.Dispose()
End Try
Dim ds1 As New DataSet
Try
Dim sExample2 as String
Dim iCount2 as Integer
'Carry out some processing with data here and write to database
Catch ex As Exception
dac.core_addLog(ex.Message) 'add a log
GoTo startOfFunction 'start again
Finally
ds1.Dispose()
End Try
'We loop round and start again
GoTo startOfFunction
End Function
Also, do I need to worry about disposing objects that are in sub routines or functions that are called from the main function, or will these be handled by garbage collection as they fall out of scope when focus switches back to main function?
Thanks
Personally I don't like the GOTO. It would be better to do
dim Done as boolean=false
While not done
'put code here
'no spaghetti code; a simple loop until done=true
End While
Also I don't see you doing anything with this dataset. Just because you call Dispose() doesn't necessarily mean the memory will decrease. It just means the garbage collector will do so at a indeterminable time. As an aside the use of case in a Java fashion is not usually done in VB. Most people use CamelCase. Capitalize the first letter of all words and not the others.
I don't see anything else that grabs me, of course this is an example and not the real code.
dim Done as boolean=false
While not done
'put code here
'no spaghetti code; a simple loop until done=true
End While
Also I don't see you doing anything with this dataset. Just because you call Dispose() doesn't necessarily mean the memory will decrease. It just means the garbage collector will do so at a indeterminable time. As an aside the use of case in a Java fashion is not usually done in VB. Most people use CamelCase. Capitalize the first letter of all words and not the others.
I don't see anything else that grabs me, of course this is an example and not the real code.
I also wonder why instead of:
If dac.core_dataToProcess > 1 Then
'There is no data to process
Thread.Sleep(5000) 'sleep for 5 seconds and start again
GoTo startOfFunction
End If
why not have
While dac.core_dataToProcess > 1
'no data to process
Thread.Sleep(5000)
End While
But I wonder the thread safety involved in checking dac.core_dataToProcess as well.
If dac.core_dataToProcess > 1 Then
'There is no data to process
Thread.Sleep(5000) 'sleep for 5 seconds and start again
GoTo startOfFunction
End If
why not have
While dac.core_dataToProcess > 1
'no data to process
Thread.Sleep(5000)
End While
But I wonder the thread safety involved in checking dac.core_dataToProcess as well.
Try setting the datasets = nothing as well hoping that this will cause the GC to collect these. Also handle the objects in other subs similarly.
You can also force GC to collect garbage. I wouldn't do it every time though.
http://www.vb-helper.com/howto_net_force_gc.html
Maybe you could have a counter that gets incremented after the
Thread.Sleep(5000) and once that is equal to 36 (3minutes have passed) or whatever time you come up with, set the counter to 0 and force GC to collect(see the link) and let it wait another N minutes before doing it again.
http://www.vb-helper.com/howto_net_force_gc.html
Maybe you could have a counter that gets incremented after the
Thread.Sleep(5000) and once that is equal to 36 (3minutes have passed) or whatever time you come up with, set the counter to 0 and force GC to collect(see the link) and let it wait another N minutes before doing it again.
ASKER
Hi,
Thanks for all the comments. I was hoping for some conclusive information, but everything seems to be people's views and opinions. People saying that they don't like goto etc are not that helpful unless they can back that up with a reason while it may be affecting memory problems or if it's just personal preference.
I need to know things like, if I declare a string variable or an integer, do I need to set that nothing before I loop and use the code again to reduce memory leaks? Should I dispose the datasets/datatables or should I set =Nothing, or both, are there any other types of objects that need special procedures to eliminate memory from never being released and leaking?
SStory: The code example I gave was off the top of my head, I know it didn't do anything with the dataset, I added a comment to indicate where I would do something in real application.
Are there any other typical reasons for memory leaking in .net, common mistakes people make? Or any methods to tackle this from the other end and monitor the service and detect where the memory leaks are occurring?
Many thanks in advance
Thanks for all the comments. I was hoping for some conclusive information, but everything seems to be people's views and opinions. People saying that they don't like goto etc are not that helpful unless they can back that up with a reason while it may be affecting memory problems or if it's just personal preference.
I need to know things like, if I declare a string variable or an integer, do I need to set that nothing before I loop and use the code again to reduce memory leaks? Should I dispose the datasets/datatables or should I set =Nothing, or both, are there any other types of objects that need special procedures to eliminate memory from never being released and leaking?
SStory: The code example I gave was off the top of my head, I know it didn't do anything with the dataset, I added a comment to indicate where I would do something in real application.
Are there any other typical reasons for memory leaking in .net, common mistakes people make? Or any methods to tackle this from the other end and monitor the service and detect where the memory leaks are occurring?
Many thanks in advance
Well, sorry you don't like opinions.
Anyhow, I was trying to tell you how to force GC to collect which may show that memory isn't really leaking, just not getting collected as you think it should. I have noticed this in dotnet apps. They seem to be using a horrendous amount of memory when a lot of times the memory is available to be reclaimed, but only when needed. GC's collect say, do it already ASAP.
From the code you've shown I see no reason for a memory leak. If you are accessing unsafe code, GDI objects or something like that then you will have to use unsafe methods to release the memory or it will cause leaks. Other than that, generally put a dispose in the finally part for appropriate objects and you shouldn't have leaks.
So my point was do you really have a leak or is it just lazy GC on collecting?
Anyhow, I was trying to tell you how to force GC to collect which may show that memory isn't really leaking, just not getting collected as you think it should. I have noticed this in dotnet apps. They seem to be using a horrendous amount of memory when a lot of times the memory is available to be reclaimed, but only when needed. GC's collect say, do it already ASAP.
From the code you've shown I see no reason for a memory leak. If you are accessing unsafe code, GDI objects or something like that then you will have to use unsafe methods to release the memory or it will cause leaks. Other than that, generally put a dispose in the finally part for appropriate objects and you shouldn't have leaks.
So my point was do you really have a leak or is it just lazy GC on collecting?
ASKER
Well, sorry you don't like opinions.
Sorry, I didn't mean it to sound like that. I was just trying to say that I know that people will have different opinions and if I followed them all then I'd be treading on my own toes before too long, so it's good to know what the "right" way to do things is.
You reference to forcing garbage collection is one that I may try, but I've just heard for so long that manual manual garbage collection is a bad idea but maybe this is one of those situations where .net doesn't know best and manual gc may help. If I force gc in one thread, does that apply to all threads within the application or just that individual thread? (sorry if that's a stupid question)
Thanks
No I think that is a good question. I really don't know the answer to it either. I'm not sure you should or shouldn't force it, but doing so even temporarily, if it has a different outcome might tell you that you think you have a leak and don't really. I don't know if they've fixed the way these get reported in all OSs or not.
>Should I dispose the datasets/datatables or should I set =Nothing, or both
http:#a37740142
Especially dispose off the datatables and datasets as these can consume a lot of memory depending on the number of rows.
http:#a37743976
Try setting the datasets = nothing as well hoping that this will cause the GC to collect these. Also handle the objects in other subs similarly.
Did you read all the comments?
http:#a37740142
Especially dispose off the datatables and datasets as these can consume a lot of memory depending on the number of rows.
http:#a37743976
Try setting the datasets = nothing as well hoping that this will cause the GC to collect these. Also handle the objects in other subs similarly.
Did you read all the comments?
ASKER
Did you read all the comments?
Hi, yes I did read all the comments but before I go through the the whole application to try to isolate areas where we can prevent potential memory leaks, I wanted to get confirmation on what I should be focussing on.
Different people said to dispose of the dataset, and another to set = nothing, I was just wondering if anyone knew conclusively whether both were necessary or not, and, if they are, then in what order. If it is a case that no-one knows for sure then that's fine and I'll just have to try and apply both and see if that solves the issues, but as it's a reasonably large task to go through the entire service code to identify I just want to make sure we have a plan of action based on the most accurate advice we can get.
Hope that makes a little more sense, it wasn't that I was not bothering to read the comments.
Cheers
SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER CERTIFIED SOLUTION
THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
ASKER
Well you could try one of these:
http://memprofiler.com/
http://www.codeproject.com/Articles/19490/Memory-Leak-Detection-in-NET
Thanks, will try those also.
x="test" & x
in a loop, or something like that, you are creating a lot of string objects. They will get collected eventually by the garbage collector, but you can't know when.
You could avoid some of that by using a stringbuilder object.
You can also use something like String.Format("test {0}",x)
It would be hard to determine your leak with no code.