asked on

Huge memory requirments

Hi.

I'm writting a C program which uses up piles of memory. Currently it fails to allocate the memory I need (calling calloc returns with a null pointer). The thing is I think I *have* the actual memory requested. I use a Win2000 dual 1MHz station with 2Gig physical RAM and adaquate HD space. I'm writting in VC++ ver6 prof. Edition.

Here is my question: Can anyone suggest what I can do to make sure the program actually takes advantage of the full memory available? Again, although I use up a lot of mem, I am pretty sure there is still much left available which the programm is not using. Can this be true? If so, what should I do?

Frost

nebeker

For Windows 2000, I believe each process is limited to 2GB of memory. How much memory are you trying to allocate? Could you post a few lines of code that shows where the failed allocation is happening?

Also, what do you mean "make sure the program takes advantage of the full memory available" ??? Do you want to query the OS to find out how much memory is installed, and allocate a fixed percentage of it?

garboua

well, I don't know much about w2k, but if you need more memory then you have, or could obtain then you can use pointer all over and then allocate and delete as needed.

if nebeke is correct, then you have a problem. 2bg of memory is a whole LOT of memory. unless you are processing a very long segment of a graphical input, and I mean hours worth of input, You have a memory leak in your program.
your source code might help

John Mc Hale

Have you considered writing the program in C++ and using the operator new to allocate memory. This may be more efficient that the older calloc, realloc etc. library functions.

nebeker

fredthered -

There are no efficiencies gained by using "new" vs. malloc, unless you intend on writing your own new handler....

Frost_Byte

ASKER

I'm an old timer, used to coding for DOS n'such. So my initial guess is that something is chocking me other than the actuall physical memory size. The OS, the compiler etc. I want to make sure that is not the case (say by asking you guys).

Posting code will be a problem - its a pretty large program, that has been "evolving" some time now, worked on by several people in the academy (you get the picture).

I don't have the exact numbers (how much mem exactly I'm trying to allocate) but I know its way less than what I have. This since calloc fails when I try to allocate memory to copy a certain array. The new array is as large as the existing one. Task manager tells me I still have way memory left, but it still fails.

Anything?

Paul Maker

is it a dos program or a win32 application

Frost_Byte

ASKER

Its a console program. Only includes stuff like stdio and stdlib, no windows.h etc.

Paul Maker

what are you compiling it with?

Frost_Byte

ASKER

VC++ ver 6. Everything is pretty much default in the settings.

Paul Maker

strange... can you reproduce this behaviour with a simple console application that just loops and allocates memory..

oleber

Can't you work arround this problem.
Cann't you send some information for hard-disk?

If nebeker is correct, did you see how many Virtual Memory you are using?

nebeker

Just by chance, are you using a signed integer to hold the amount of memory being requested? Depending on what type of variable (int, short int, etc.) you're using, and how much memory you're asking for, you might have overflowed the capacity of that variable -- which would mean you're asking for a "negative" amount of memory :)

Frost_Byte

ASKER

nebeker : interesting idea, but no. I allocate the same amount of memory twice (this happens when I make a copy of a large _existing_ array). The first time it works, only fails on the second try. That and I calloc an array of length smaller than 10mil of large structs. The command is ...calloc(N * sizeof(BigStruct)). No signed\unsigned issues, as far as I can tell.

oleber : Have around 5gig swap file. I think that should be enough for my needs.

makerp : I'll have to try that but it'll take me time. Will get back to you on that today-tomorrow.

I understand that there is no general known issue with memory limitations, right? There is no hidden switch labeled "let me use everything you got" that I haven't been pushing?

weicco

Have tried to call GetLastError / FormatMessage after calloc returns NULL, what does it say?

Hmm... I had wild idea. Try to allocate memory with CreateFileMapping.

ASKER CERTIFIED SOLUTION

jonnin

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

CSLIU

From 386 to IA32, processors are 32-bit.
The maximum physical addressing space is 4GB in 386 protected mode.

http://support.microsoft.com/support/kb/articles/Q268/3/63.asp

http://support.microsoft.com/support/kb/articles/Q283/0/37.ASP

C code:
/* an example */
#include <stdlib.h>
#include <stdio.h>
int main()
{
void *largeP;
unsigned int uSize; /* unsigned int is 4bytes(32-bit) in VC++6.0 */

uSize = 0x80000000 - 1; /* 2G-1 */

largeP = malloc( uSize );
if( largeP == NULL )
{
printf("Ok");
free( largeP );
}
else
printf("Falied");

return 0;
}

weicco

"The best fix is not to use dynamic memory anywhere you don't have to (won't fit on the stack, or other reasons like run time determined size, etc)."

Doesn't malloc use heap not stack?

Frost_Byte

ASKER

Well I downloaded some mem-checking utils, not rambooster, but FreeRam XP pro (also free). I still haven't solved the problem but I now know that I have a very fragmented memory which I guess is causing all this. I am, in fact, doing tons of allocating, freeing of small mem fragments. Looks like I have no way of avoiding actual recoding.

Thanks all,
Frost

weicco

Little comment... If you need to allocate & realloca & free lots of small memory fragments I suggest you write your own memory manager.