Solved

System Out of Memory Exception

Posted on 2009-05-19
28
3,353 Views
Last Modified: 2012-05-07
Hi

I am having an application running on production server. There are around 100 domains running on that server though we are not getting muhc traffic right now we are getting 2,500 unique Visitors. Now twice a day I get system out of memory exception ..and by applying so many things ..I am unable to solve it .. The structure of my application is 3 Tier .. and following is the code for classes I am using ...I want to know is anything in classes I am doing wrong which is causing it. Please see the code below. This is just a sample class.

For further information. I am using Idisposable for all my classes and also using supressfinalize in dispose method. For access all classes I am USING keyword .. May be i missed it for some classes ..but I don't think it really affect ?

Also I have checked my all SQL connections are getting closed properly. using sp_who command .. generally I see around 2500 connections among 7 databases which I am using in my application ..as my application is a portal.
using System;

using System.Data;

using System.Collections;
 

/// <summary>

/// Summary description for clsCarPayments

/// </summary>

public class clsCarPayments : IDisposable

{

    #region --Declare Variables --

    int intAgentId = 0;

    Hashtable HTable = new Hashtable();

    clsCarDb objDatabase;

    #endregion
 
 

    #region --Declare Properties --
 

  

    public int AgentId

    {

        get { return intAgentId; }

        set { intAgentId = value; }

    }
 

    #endregion

    public clsCarPayments()

	{

		//

		// TODO: Add constructor logic here

		//

	}
 

    //~clsCarPayments()

    //{

    //    

    //}

    public void Dispose()

    {

        GC.SuppressFinalize(this);

    }
 

    public DataTable getAgentPayments()

    {

        using (objDatabase = new clsCarDb())

        {

            HTable.Clear();

            HTable.Add("AgentId", intAgentId.ToString());

            return objDatabase.procFetchRecord_DS("uspSelectAgentPayments", HTable).Tables[0];
 

        }

    }

}
 
 

------------------------------------------------------ Following is the database class function------
 

 public DataSet procFetchRecord_DS(string procname, Hashtable Htab)

    {

        try

        {
 

            sqlCmd = new SqlCommand();

            sqladp = new SqlDataAdapter();

            sqlds = new DataSet();
 

            sqlCmd.Connection = con;

            sqlCmd.CommandType = CommandType.StoredProcedure;

            sqlCmd.CommandText = procname;

            IDictionaryEnumerator ObjEnumerator = Htab.GetEnumerator();
 

            while (ObjEnumerator.MoveNext())

            {

                sqlCmd.Parameters.Add(new SqlParameter("@" + (ObjEnumerator.Key).ToString(), ObjEnumerator.Value));

            }

            sqladp.SelectCommand = sqlCmd;

            sqladp.Fill(sqlds);
 

        }

        catch (Exception ex)

        {

            sRetVal = ex.Message.ToString();

            //     sqlTran.Rollback();

        }

        finally

        {

            closeDBConnection();

        }

        return sqlds;
 

    }

Open in new window

0
Comment
Question by:sachinjain_aspnet
  • 13
  • 9
  • 5
  • +1
28 Comments
 
LVL 11

Expert Comment

by:Muhammad Ousama Ghazali
Comment Utility
Although the version of .NET you are usign is not mentioned, but if it is .NET 1.1. in one or more websites hosted, please consider seeing these two official fixes for your problem:
http://support.microsoft.com/kb/888419
http://support.microsoft.com/kb/825680
0
 
LVL 12

Expert Comment

by:GuitarRich
Comment Utility
the class looks ok to me - what are the machine specs?
also - how do you have IIS setup? how many concurrant users do you have? If you profile the server using perfmon can you see the memory being released properly or does it steadily increase?
You might want to think about how often the application pool recyles - do it more often to release some memory??
0
 

Author Comment

by:sachinjain_aspnet
Comment Utility
Sorry not to mention version ..I am using .net 2.0
0
 

Author Comment

by:sachinjain_aspnet
Comment Utility
I am recycling worker process at every 12 hour. This resolve my problem but I don't want to do that ..I want to know actual reason why it is happening. Just recycling worker process doesn't give actual solution of this
0
 
LVL 12

Expert Comment

by:GuitarRich
Comment Utility
what about my other questions:
what are the machine specs?
also - how do you have IIS setup? how many concurrant users do you have? If you profile the server using perfmon can you see the memory being released properly or does it steadily increase?
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
You have something very odd in your code. You are asking why you use up a lot of memory and then you state this:

For further information. I am using Idisposable for all my classes and also using supressfinalize in dispose method. For access all classes I am USING keyword .. May be i missed it for some classes ..but I don't think it really affect ?
which is effectively saying "I use suppressfinalize to prevent my classes to be garbage collected". If you do that, the objects having that, will never be freed. Unless there is a very specific reason to use IDisposable i.e., using unmanaged sources that need to be finalized) you do not need it. And if you use IDisposable (or a class that uses it), there are only very few reasons to need SuppressFinalize.

I suggest removing all those lines and try again.

PS: testing for the memory usage is not enough. You may run out of memory before you system runs out of memory. That is because the CLR in ASP.NET uses its own pool. You can trace that using Performance Monitor.

-- Abel --
0
 

Author Comment

by:sachinjain_aspnet
Comment Utility
GuitarRich

System Configuration are as follows

CPU INTEL Pentium-IV x86 Family 15 Model 4 Stepping 3 ~3600MHz
Version psa v8.4.0_build20080505.00 os_Windows 2003/2008
OS Microsoft Windows 5.2;build-3790;sp2.0;suite272;product3
RAM - 2GB

Regarding concurrent users - I don't know exactly as I don't know how to watch it. But you can take a guess on average we are getting 3,000 hits per site per day NOT UNIQUE VISITORS

0
 
LVL 12

Expert Comment

by:GuitarRich
Comment Utility
good call abel - I missed that. Try that suggestion
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
This explains my point more thoroughly: http://stackoverflow.com/questions/151051/when-should-i-use-gc-suppressfinalize

More to the point: you should call SuppressFinalize only AFTER everything is freed. In your case, there's no reason to use IDisposable (you do not use unmanaged resources) so you should not need to use SuppressFinalize either.

Remove all cases where you implement IDisposable, apart from those situations where you really need to do cleanup (close open file handles for instance). In the cases where you really need to do cleanup, remove SuppressFinalize.

You say you use using-statements. That's good. You should use them on classes that expose the IDisposable interface (streams come to mind). It is not always necessary, but it is better to be safe then sorry. Only rarely you will need it for your own objects.

-- Abel --
0
 

Author Comment

by:sachinjain_aspnet
Comment Utility
So Abel

According to you ..I should remove I disposable from all classes .. even database class as well ?
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
(tx, GuitarRich)

On another angle: while you are fixing your code, you don't want the current site to run out of memory all the time. Check the following (if you haven't already done so) and increase (see screenshot, max virt mem),

Depending on the occurrence frequency, you can configure IIS to restart your web application on fixed times. This is only a temporary workaround until your code is fixed. But it will prevent your visitors to move away. If you have a StateServer, there sessions will remain, otherwise, your users need to logon after restart. It is not a machine restart (see screenshot, recycle worker processes in minutes).

This screen looks different on IIS6/7, but the idea is the same.

ScreenShot293.png
0
 
LVL 12

Expert Comment

by:GuitarRich
Comment Utility
another option might be to use a web garden - give the application pool a number of worker processes to share out the memory load - you would definitely then need the session setting out of process tho - it might be less work just to change your classes as abel has suggested.
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
> even database class as well ?

what do you mean with a database class? If you mean the entities that represent your data, then yes. If you mean the classes that use your database any which way then yes (but keep the using-statement for your sqlconnection etc). If you mean classes that act as a database by exposing or inheriting from existing database classes then yes.

Only if you created your own ODBC driver wrapper or something becaue you access some exotic database, you may access unmanaged code or resources with your class and keep pointers to them. Think about C++ interfaces or that sort of thing. If the CLR cannot do it automatically, you need to help. But very rarely I see this in practice.

Now there may be cases where you keep a reference to an object that exposes IDisposable itself, and you keep them in your member variables. In those cases, you may also implement the IDisposable interface, but only strictly to cleanup (not yourself, but the references). Here's how such frame should look like: http://www.codeguru.com/csharp/.net/net_general/debugginganderrorhandling/article.php/c14173
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
By now it is time to tell a little about what IDisposable actually does. It doesn't do much on its own. If it is used properly in try/finalize blocks or within a using-block, then it will prevent memory leaks (at least, that's the idea).

Even if you don't use using-blocks and you don't use try/finalize, the memory will be freed in most cases. Only when exceptions occur, memory may not be freed correctly.

Applications that are sloppy with try/finalize and using-blocks see a slow memory increase sometimes, but if no errors occur, nothing wrong will happen, really.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:sachinjain_aspnet
Comment Utility
thanks abel .. now I will remove idisposable and suppresfinalize from all classes .. though i have 100 of classes ...it take some time ..and its reaction need be watched which will take 2-3 weeks ..if everything okay ..otherwise i will come back again
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
Ok, good luck with the cleanup process. Be careful about those situations where you do need them, please read the two links carefully, I chose them for there clarity and brevity.

Note that it never hurts to use a using-block, and it often doesn't hurt to use a IDisposable implementation, but the extra work is void  (see my last link on how a good implementation should look in those cases where it matters) but SuppressFinalize is probably your real problem here (and do notice, too, that there may be other forces at play that push your memory. I.e., a session that has a long timeout period, or large objects attached to the session, cache or globals etc).

While you are working on it, you can (or not) choose to use the tips I gave you about IIS to prevent your site to show Memory errors to your end-users.
0
 

Author Comment

by:sachinjain_aspnet
Comment Utility
Abel

I am already using web garden and worker process recycle so that error doesn't come. But I also need to clean up my code.

Regarding the session on every request there are 6-7 session variables made. I have session time out to 20 minutes in web.config file. Login sections are seperate for which we create session/cookies. I don't think they will hurt do they ?

SupressFinalize i will remove from all my classes but will use in database class to make sure that connection of current object get closed every time after calling dispose method
0
 
LVL 39

Accepted Solution

by:
abel earned 250 total points
Comment Utility
> but will use in database class to make sure that connection of current object get closed every time after calling dispose method

I hope you mean to say that that's the place where you do not use SuppressFinalize, because that's precisely a place where you should not use SuppressFinalize, and it is precisely what it does not do, instead, it prevents cleaning up.

You say you are using a database connection. If it is wrapped around a using-block, you do not need to call Dispose or inherit IDisposable. What you certainly shouldn't do is to call SuppressFinalize, because that tells the CLR garbage collection that finalization is already done, and that the GC should not try to dispose or finalize anymore. In other words: it prevents finalization / destruction / freeing up memory.

If your database class is using a field that is holding a reference to the database connection, than you probably use a thread-safe singleton class, which will keep the connection alive to prevent unnecessary connections to be created. A thread-safe singleton does not need to implement IDispose.

If you do not have a singleton and you create the database connection for each request again, you should either wrap it in a using-block, or, if that cannot be done, implement the IDisposable interface like in that link I sent you. The line with SuppressFinalize there is put after the repeated call to its internal Dispose(bool) method and after all resources are freed, inside the Dispose() method. The example you show above does the opposite: it prevents the garbage collector to free up resources.

Be very secure when you want to work with IDisposable and its friends. Making coding mistakes there will lead to leaking memory. If you can stick to just use using-blocks, you are on the safe side and you do not need to worry about the subtleties involved.

-- Abel --

PS: in short, I believe your code should not have any call to SuppressFinalize, ever. The possibility of using it wrongly is just to large. And not using it will not result in memory loss.
0
 
LVL 39

Assisted Solution

by:abel
abel earned 250 total points
Comment Utility
On sessions: it really depends on how large the sessions are. In a recent project, it seemed reasonable to have the search results (a product list) stored in the session. After testing we found that the search result list was about 2MB (memory, occupied by 700 entity objects). With average 500 users online at any given time, we decided it too risky (500 x 2MB = 1GB) and resorted to another technique.

So, you can have only one session variable, and still have gigabytes of memory occupied. It all depends on how big these session variables really are. As a rule of thumb: check the serialized size of the variables, if they are more then just strings or integers, which you can by storing it to disk. But that's only a subject of investigation when the current approach does not yield any results.
0
 

Author Comment

by:sachinjain_aspnet
Comment Utility
Hi Abel

Can you give me your email ? I think its best I email you some of my sample files of project and you can tell me flaw in coding ?
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
I wouldn't mind doing so, but if I would, I am violating EE guidelines. My email address is in my profile, normally it should only be used when professional services are requested (which I wouldn't mind you requesting, of course, but in which way it is probably easier to setup a VPN bridge of some sort ;-)
0
 

Author Comment

by:sachinjain_aspnet
Comment Utility
well Abel, I just want to send code to see you If it resolve my problem i won't mind paying but my error shud be resolve at last and i hope thats not much ...coz i am a normal developer :) i will be sending you email from my gmail account .. i hope you will get those files
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
Yes, I've seen the files, thanks. The main thing I looked at was your database class, I didn't look at the other classes. Inside the database class you have a properly implemented IDisposable interface, but (possibly amongst other things) it does not call Dispose for the "con" member (the connection object), which will mean in practice that the connection pool and some other stuff may stick around.

I also saw something odd in the  constructor: when the connection is closed, it opens it, when it is open, it will close it. It will never be open upon creating, so this code is moot, but it doesn't harm either. Didn't check futher, though.

PS: note that calling .Dispose() will always also call .Close() for any classes in the CLR, if a Close() is applicable.
0
 
LVL 39

Assisted Solution

by:abel
abel earned 250 total points
Comment Utility
Looking slightly further through your code, I see intermittent connection .Close() inside finally blocks, I see "new SqlCommand" where the related object (even though it is a member, but it is newed each time again) is not Disposed. Example of both of these errors is in checkRecordExistance.

Examples of not Disposing newly made objects (they are remade on every call, so they must be re-disposed at the end of every call) are in FillSubDetail_With_ParamPageIndex, FillSubDetail_With_Param, RetriveAllRecords (SqlCommand, SqlDataAdapter, DataSet) and so on.

Many of these commands are not typed, which means they return an object that is late bound. There can be a design decision to do so, but the result is unpredictable, because you loose the IDisposable interface information. Take the code below, the following needs to be corrected:

  • the SqlCommand object should be in a using-block, the sqlCmd should not be a member variable, or you should only initialize it inside the constructor. Neither are true. Use a local variable and put a using-block around it.
  • the method is not typed, there's no way the compiler can know whether to use IDisposable on the returned object. You cannot use a using-block for that.
  • you return an ExecuteReader, which itself is an SqlDataReader and implements IDisposable. There is no using-block present. The best thing to do is assign it to a local variable in a using-block and return the results from the block.
  • that you call .Dispose on the sqlCmd member in your own Dispose method, is void, because by that time the member has seen many assignments that were never Disposed of.
I chose the below example because it is very simple and illustrates the main problem with the code (and it doesn't show anything private). Not just one or two objects are not diposed, but literally hundreds of them. That single missing call to Dispose is not the problem, this is likely your real problem and is all through the code.

My advice: go meticulously careful through all your code, check for any assignments of variables with IDisposable and make these variables local and put a using-block around them. This will impact your performance (don't worry, it will probably be faster then it is now) as compared to the most ideal design, but it is the best bet for now. It will be a lot of work, that's for sure (sorry about giving bad news...)

-- Abel --

PS: choosing an ORM when you do your next project will release from literally all these problems and all you are doing henceforth is playing with the objects and lists, which is much easier, really.

public SqlDataReader RetriveAllRecords(string tableName, string searchColumns)

{

    sqlCmd = new SqlCommand();

    sqlCmd.Connection = con;

    sqlCmd.CommandText = "select " + searchColumns + " from " + tableName;

    return sqlCmd.ExecuteReader();

}

Open in new window

0
 
LVL 12

Expert Comment

by:GuitarRich
Comment Utility
I think more points should be on offer here! definitely going above & beyond the call of duty here abel!
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
That's nice, but it's all in the service, so far ;-)
0
 
LVL 39

Expert Comment

by:abel
Comment Utility
In the line of "what to read to understand IDisposable" and implementing it oneself better, this is yet another excellent read. Check out the code of the first answer (8 votes at the time of this writing): http://stackoverflow.com/questions/792660/is-gc-suppressfinalize-guaranteed

I uses just a slightly different wording that might make the whole process more clear to you.

-- Abel --
0
 

Author Closing Comment

by:sachinjain_aspnet
Comment Utility
thanks a lot
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Suggested Solutions

International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now