Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

heap fragmentation

Posted on 2007-10-03
12
Medium Priority
?
689 Views
Last Modified: 2013-12-04
Hi,

I'm running into a problem allocating memory on winXP. Memory is being allocated in some threads, which looks like:

     int ThreadFunction()
     {
           PauseThread();
           Allocate();
     }

On my test case, I have 4 threads open which all execute that same function - 902mb are being used in the system at this point - before this point, the memory usage rose to as high as 1.5 gb. Which ever thread of the 4 waiting I choose to let continue first always fails Allocate() with a general memory allocation error (bad alloc). Since the thread dies, it releases its other memory and the other three threads all get through it ok.

I'm guessing this is a fragmentation problem? Is there anyway to compact the heap? Is this an OS specific thing? Is there anyway to specifically check that this is a fragmentation problem?

Thanks
0
Comment
Question by:DJ_AM_Juicebox
12 Comments
 
LVL 13

Expert Comment

by:josgood
ID: 20008733
WinXP provides a low-fragmentation heap, which may be of service.  See
    http://msdn2.microsoft.com/en-us/library/aa366750.aspx


0
 
LVL 4

Expert Comment

by:Ichijo
ID: 20008737
I believe heap fragmentation is independent of the operating system you use, and you would need to write your own memory allocator to get around it.
0
 
LVL 13

Expert Comment

by:josgood
ID: 20008780
Creating your own memory pool is another solution, as Ichijo points out.  That might be a good solution for you, since all allocation is performed by Allocate() and (I assume) the allocation sizes are all the same -- or at least over a known range.

However, I wonder if we aren't going in the wrong direction.  Someone has to be freeing memory along the way, otherwise you will of course run out of memory.

Do you know for sure that allocations are being freed?
Are you using raw pointers (char *myPtr) instead of smart pointers?
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 13

Expert Comment

by:josgood
ID: 20008804
Following up on my smart pointer comment, Wikipedia has a nice article on auto_ptr
http://en.wikipedia.org/wiki/Auto_ptr

Just make sure you aren't storing auto_ptrs into STL collections (auto_ptr carries copy semantics)
0
 

Author Comment

by:DJ_AM_Juicebox
ID: 20008957
I'm sure everything is getting freed ok. I'm allocating huge volume buffers (just really big 1D arrays), probably like 80mb each, then low resolution versions of them too, anyway it gets to be about 1.5gb at some point.

I want to avoid using any of the win32 specific memory allocation routines.

Well, I guess I just wanted to know if there was some c++ HeapCompact() function, but I don't think there is.

I guess I just need to allocate everything up front to make sure these huge buffers are all in one segment instead of all over the place.

0
 
LVL 13

Expert Comment

by:josgood
ID: 20009014
I'm not aware of any library that contains a HeapCompact().  

I think you're on the right track.  If you preallocate your buffers, then you have control.  As you imply, that will tend to give you unfragmented memory.

I suggest allocating larger buffers first, but you've probably already thought of that.

Good luck!


0
 
LVL 86

Accepted Solution

by:
jkr earned 800 total points
ID: 20009056
>>Well, I guess I just wanted to know if there was some c++ HeapCompact()
>>function

HeapCompact(GetProcessHeap(),0);

will do that for each app on each MS-OS, C++ or any other language. The C++ memory allocation routines will end up at Win32 APIs at some point anyway.
0
 
LVL 13

Expert Comment

by:josgood
ID: 20009321
Very cool, jkr!  You are the man.
0
 
LVL 86

Expert Comment

by:jkr
ID: 20009920
BTW, if there are private heaps allocated via 'HeapCreate()', you can still

DWORD dwHeaps = GetProcessHeaps(0,NULL);

HANDLE pHeaps = new HANDLE[dwHeaps];

GetProcessHeaps(dwHeaps,pHeaps);

for(DWORD dw = 0; dw < dwHeaps; ++dw) {

  HeapCompact(pHeaps[dw],0);

}
0
 
LVL 86

Expert Comment

by:jkr
ID: 20011393
Oops, I forgot to add a hearty

delete [] pHeaps;

in the above...
0
 
LVL 39

Assisted Solution

by:itsmeandnobodyelse
itsmeandnobodyelse earned 800 total points
ID: 20012908
I do not know whether HeapCompact can solve your problems. HeapCompact can *not* move any memory currently in use. So, the condition that it can add a significant share to defragmentation is that your prog before allocated a lot of small storage which now can be coalesced to bigger pieces *or* that there is a significant amount of free memeory which waits on committing to be freed. Actually I cannot see in your scenario above that either of these might be the case.

If your prog got 1.5 gb of storage (I assume you've 2 GB physical memory in total?), then it is a very big amount. You may argue that your prog gets virtual memory which can be greater than physical memory but that is true only for rare cases where your process nearly is alone and where both physical memory and pagefile threfore can be used by your prog nearly exclusively. In real life your process shares the storage with 30 to 50 other processes where a process hardly can (and should) get more than the half of the total especially if it needs big contiguous chunks. If you really need that you may consider using a ram disk, where the reservation is made at a very early time and where for that some of the restrictions may not apply. You then could think of using 3gb or 4 gb of physical memory though it still will depend on your hardware and os version whether you can make use of it.


Regards, Alex
0
 
LVL 13

Assisted Solution

by:josgood
josgood earned 400 total points
ID: 20013139
itsmeandnobodyelse makes a very good point about available physical memory and OS version.

I did a small study about a year ago on the equipment we sell.  We have 4GB of physical memory and are running WinXP.  It turns out the memory address layout has an effect also.  On some boards, WinXP-32 made just over 1GB of physical memory available to application space, on others we got as much as 1.6GB available to application space.  It turned out that WinXP-32 had difficulties dealing with gaps in physical memory space.

Win 2003 Server and WinXP-64 did not have these problem -- they both saw 4GB total and assigned 2GB each to system and application space.  64-bit Ubuntu also saw 4GB.

I suggest you install 4GB of memory (minimum) and use WinXP-64 or Win 2003 Server.  Your 32-bit application can remain unchanged.
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

For most people, the WrapPanel seems like a magic when they switch from WinForms to WPF. Most of us will think that the code that is used to write a control like that would be difficult. However, most of the work is done by the WPF engine, and the W…
Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
The viewer will learn how to user default arguments when defining functions. This method of defining functions will be contrasted with the non-default-argument of defining functions.
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.

564 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question