Memory leakage problem

Posted on 2005-05-11
Last Modified: 2012-06-21
We are having a C++ converter which converts messages into a specific format based on the input.
The problem is whenever this program is called with large number of records, it is taking up huge memory ( > 500 MB) and later on crashes.
There is no issue if the input data is less.

I've identified some point in the code where it might go wrong, can anyone pls help in this issue ---

                Msg(szCallerID, "Create Context structure failed");
                return FAILURE;
        memset(pGConvObjs, 0, sizeof(CONVERTER_OBJS));

iRet = SetTemplate(pGConvObjs);

In SetTemplate function, the error is being thrown when trying to load dynamic libraries.

In header file, we have defined --
    char                                szUniqueName[C_MAX_PATHNAME];
        CDataRep*                       pGDataRep;
        CEnvironment*           pGEnv;
        CSysLog*                        pGErr;
        CProfile*                       pGProfile;
        CDBAccess*                      pGDBAccess;
        CTemplateBuilder*       pGInTmpt;
        CTemplateBuilder*       pGOutTmpt;
    CGVarFuncTable*             pGVarFunc;
        CInBound*                       pGInBound;
        COutBound*                      pGOutBound;
        char*                           pBindVar;
        int                                     iNumBindVar;
    HINSTANCE                   outPgxHandle;
    HINSTANCE                   inPgxHandle;

Is there any problem, in initializing the memory with the above memset(),
or should it be done like this ---
memset(pGConvObjs, '\0', sizeof(pGConvObjs));

Pls advise.
Question by:prasen120998
    LVL 8

    Expert Comment

    there are many pointers in the structure CONVERTER_OBJS, did you verify that they are getting destoryed?

    You can consider adding a destructor to your structure.


    Accepted Solution

    [*] DLL Loading problem:
    A. Usage of large amount of memory e.g. 500Mb may cause the heap to be fragmented.
    B. Loading a DLL into memory require certain amount of continues memory.

    Taking in mind what was just said, large amont of fragmented memory may cause the heap to have no free pages to load the DLL, this will cause LoadLibrary to fail.

    Resolution: memory allocations should occur as less as possible ( e.g. call new as less as possible ), rather allocate a stack of memory blocks ( or structures ) such as CONVERTER_OBJS on startup and use them when needed, keep them allocated until the application terminates, if not enough blocks are available allocate new blocks and add them to the stack, try to make as less as possible redundant new-delete calls.
    Refer to the lookasidelist mechanism for better explanation of the mechanism I have described ( search the DDK or the net for 'ExInitializeNPagedLookasideList' ).
    - using the approach I have described may solve the LoadLibrary problem but will not neccesarly solve the memory consumption problem.

    [*] Concerning the memset usage:
    The way you set the memory is OK.
    using 'memset(pGConvObjs, '\0', sizeof(pGConvObjs))' will only set the first four bytes of 'pGConvObjs' to Zero as 'pGConvObjs' is a pointer and the size of any pointer is four.

    [*] Concerning the leak:
    1. Make sure that when deleting an object of type CONVERTER_OBJS you delete ( if needed ) all the referenced objects ( e.g. pGDataRep, pGEnv, ... )
    2. Make sure all of the tasks are not executed simultaneously, this will lead to large memory consumption. if this is the case, try to serialize the tasks execution e.g. execute no more then X tasks together, this could be easily implemented by a thread-pool mechanism or so...

    - Also, you can use tools such as Numegas BoundsChecker to identify the leak.

    Hope this helps,

    Author Comment

    Can you pls elaborate what you meant by adding some destructors to the structure.
    LVL 30

    Assisted Solution

    I recommend you use the name of your variable when you use memset to zero out a structure.  It's safer, because if you ever change the structure you want to use with that variable, the code will still work.
    In your case, since the variable is a pointer, you need to dereference it.

    memset(pGConvObjs, 0, sizeof(*pGConvObjs));

    In Win32 programming you can also use ZeroMemory.

    ZeroMemory(pGConvObjs, sizeof(*pGConvObjs));

    It's more explicit to the reader as what you're trying to do.

    Author Comment

    Regarding destroying the pointers in the structure, all the pointers has been deleted except for pGEnv, as it has been used globally
    LVL 30

    Expert Comment

    >>Regarding destroying the pointers in the structure, all the pointers has been deleted except for pGEnv, as it has been used globally

    You can try using _CrtDumpMemoryLeaks function to try and determine where your leaks are at.

    Expert Comment

    I would recommand you to control the life cycle of your objects using reference count, with this concept each object manage it's own life cycle, hence, as long as the object is used it will not be removed from memory AND it will be removed from memory at the minute it is not bing used any more.

    To enable reference counting to your objects do the following:
    [1] Add a local reference count variable to each of your objects ( e.g. CDataRep, CEnvironment, ... ).
    [2] Add each of the objects the following methods:
    class MyObj
        LONG m_lRefCnt;
        inline LONG AddRef() { return InterlockedIncrement(&m_lRefCnt); }
        LONG Release()
            LONG lRef = InterlockedDecrement(&m_lRefCnt);
            if(0 == lRef)
                 delete this;
            return lRef;
    [3] Upon initialization ( e.g. in the constructor or creator function ) set the ref count to zero.
    [4] Each time the object is reffered to ( or used ) by another object call AddRef and each time the object stops being used call Release.

    Using this logic will gurantee the object will NOT stay in memory whe no one use it AND will garuntee that the object will stay in memory as long as it is being used.

    In your case I would recommand using this concept with all of the types referred to by CONVERTER_OBJS with pointers, at the destructor of CONVERTER_OBJS call Release on all of these objects.
    LVL 30

    Expert Comment

    You can get that type of logic, without putting it into your class.
    Just use a reference counting smart pointer wrapper class.
    See following example:

    Use it similar to the STL std::auto_ptr smart pointer.

    In general, Smart pointers delete the object automatically when smart pointer goes out of scope, so you don't have memory leaks.

    However reference counting smart pointers delete the object when the reference counter hits zero.

    In either case, you avoid memory leaks, but the reference counting smart pointer is more efficient in general.
    LVL 11

    Assisted Solution

    It is not safe to memset something that has been newed.  If you add a class to that structure, you'll be completely screwed when you do a memset.  If you want to memset it, forget about C++, just use malloc.  If you want to use new, ininitialize the structure (which is actually class { public: ...}) in the constructor.

    Author Comment

    I've not used _CrtDumpMemoryLeaks function before, so not sure how to use this function, could you pls elaborate on this.
    how it will work in identifying the mem leaks in the code.
    LVL 4

    Expert Comment

    use resource allocation is initialization pattern

    all ur problemget solved

    u can find lot of info on net.

    LVL 14

    Expert Comment

    No comment has been added to this question in more than 21 days, so it is now classified as abandoned..
    I will leave the following recommendation for this question in the Cleanup topic area:

    Split between Nadavrub, Axter, and cup

    Any objections should be posted here in the next 4 days. After that time, the question will be closed.

    EE Cleanup Volunteer

    Featured Post

    Why You Should Analyze Threat Actor TTPs

    After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

    Join & Write a Comment

      Included as part of the C++ Standard Template Library (STL) is a collection of generic containers. Each of these containers serves a different purpose and has different pros and cons. It is often difficult to decide which container to use and …
    Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
    The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
    The viewer will learn how to clear a vector as well as how to detect empty vectors in C++.

    728 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    17 Experts available now in Live!

    Get 1:1 Help Now