We help IT Professionals succeed at work.

heap fragmentation

716 Views
Last Modified: 2013-12-04
Hi,

I'm running into a problem allocating memory on winXP. Memory is being allocated in some threads, which looks like:

     int ThreadFunction()
     {
           PauseThread();
           Allocate();
     }

On my test case, I have 4 threads open which all execute that same function - 902mb are being used in the system at this point - before this point, the memory usage rose to as high as 1.5 gb. Which ever thread of the 4 waiting I choose to let continue first always fails Allocate() with a general memory allocation error (bad alloc). Since the thread dies, it releases its other memory and the other three threads all get through it ok.

I'm guessing this is a fragmentation problem? Is there anyway to compact the heap? Is this an OS specific thing? Is there anyway to specifically check that this is a fragmentation problem?

Thanks
Comment
Watch Question

Top Expert 2007

Commented:
WinXP provides a low-fragmentation heap, which may be of service.  See
    http://msdn2.microsoft.com/en-us/library/aa366750.aspx


Commented:
I believe heap fragmentation is independent of the operating system you use, and you would need to write your own memory allocator to get around it.
Top Expert 2007

Commented:
Creating your own memory pool is another solution, as Ichijo points out.  That might be a good solution for you, since all allocation is performed by Allocate() and (I assume) the allocation sizes are all the same -- or at least over a known range.

However, I wonder if we aren't going in the wrong direction.  Someone has to be freeing memory along the way, otherwise you will of course run out of memory.

Do you know for sure that allocations are being freed?
Are you using raw pointers (char *myPtr) instead of smart pointers?
Top Expert 2007

Commented:
Following up on my smart pointer comment, Wikipedia has a nice article on auto_ptr
http://en.wikipedia.org/wiki/Auto_ptr

Just make sure you aren't storing auto_ptrs into STL collections (auto_ptr carries copy semantics)

Author

Commented:
I'm sure everything is getting freed ok. I'm allocating huge volume buffers (just really big 1D arrays), probably like 80mb each, then low resolution versions of them too, anyway it gets to be about 1.5gb at some point.

I want to avoid using any of the win32 specific memory allocation routines.

Well, I guess I just wanted to know if there was some c++ HeapCompact() function, but I don't think there is.

I guess I just need to allocate everything up front to make sure these huge buffers are all in one segment instead of all over the place.

Top Expert 2007

Commented:
I'm not aware of any library that contains a HeapCompact().  

I think you're on the right track.  If you preallocate your buffers, then you have control.  As you imply, that will tend to give you unfragmented memory.

I suggest allocating larger buffers first, but you've probably already thought of that.

Good luck!


CERTIFIED EXPERT
Top Expert 2012
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION
Top Expert 2007

Commented:
Very cool, jkr!  You are the man.
jkr
CERTIFIED EXPERT
Top Expert 2012

Commented:
BTW, if there are private heaps allocated via 'HeapCreate()', you can still

DWORD dwHeaps = GetProcessHeaps(0,NULL);

HANDLE pHeaps = new HANDLE[dwHeaps];

GetProcessHeaps(dwHeaps,pHeaps);

for(DWORD dw = 0; dw < dwHeaps; ++dw) {

  HeapCompact(pHeaps[dw],0);

}
jkr
CERTIFIED EXPERT
Top Expert 2012

Commented:
Oops, I forgot to add a hearty

delete [] pHeaps;

in the above...
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION
Top Expert 2007
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.