Solved

Huge memory requirments

Posted on 2002-07-28
19
279 Views
Last Modified: 2010-04-15
Hi.

I'm writting a C program which uses up piles of memory. Currently it fails to allocate the memory I need (calling calloc returns with a null pointer). The thing is I think I *have* the actual memory requested. I use a Win2000 dual 1MHz station with 2Gig physical RAM and adaquate HD space. I'm writting in VC++ ver6 prof. Edition.

Here is my question: Can anyone suggest what I can do to make sure the program actually takes advantage of the full memory available? Again, although I use up a lot of mem, I am pretty sure there is still much left available which the programm is not using. Can this be true? If so, what should I do?

Frost
0
Comment
Question by:Frost_Byte
  • 5
  • 3
  • 3
  • +6
19 Comments
 
LVL 5

Expert Comment

by:nebeker
Comment Utility
For Windows 2000, I believe each process is limited to 2GB of memory.  How much memory are you trying to allocate?  Could you post a few lines of code that shows where the failed allocation is happening?

Also, what do you mean "make sure the program takes advantage of the full memory available" ???   Do you want to query the OS to find out how much memory is installed, and allocate a fixed percentage of it?
0
 
LVL 5

Expert Comment

by:garboua
Comment Utility
well, I don't know much about w2k, but if you need more memory then you have, or could obtain then you can use pointer all over and then allocate and delete as needed.

if nebeke is correct, then you have a problem.  2bg of memory is a whole LOT of memory.  unless you are processing a very long segment of a graphical input, and I mean hours worth of input, You have a memory leak in your program.
your source code might help
0
 
LVL 13

Expert Comment

by:John Mc Hale
Comment Utility
Have you considered writing the program in C++ and using the operator new to allocate memory. This may be more efficient that the older calloc, realloc etc. library functions.
0
 
LVL 5

Expert Comment

by:nebeker
Comment Utility
fredthered -

There are no efficiencies gained by using "new" vs. malloc, unless you intend on writing your own new handler....
0
 

Author Comment

by:Frost_Byte
Comment Utility
I'm an old timer, used to coding for DOS n'such. So my initial guess is that something is chocking me other than the actuall physical memory size. The OS, the compiler etc. I want to make sure that is not the case (say by asking you guys).

Posting code will be a problem - its a pretty large program, that has been "evolving" some time now, worked on by several people in the academy (you get the picture).

I don't have the exact numbers (how much mem exactly I'm trying to allocate) but I know its way less than what I have. This since calloc fails when I try to allocate memory to copy a certain array. The new array is as large as the existing one. Task manager tells me I still have way memory left, but it still fails.

Anything?
0
 
LVL 10

Expert Comment

by:makerp
Comment Utility
is it a dos program or a win32 application
0
 

Author Comment

by:Frost_Byte
Comment Utility
Its a console program. Only includes stuff like stdio and stdlib, no windows.h etc.
0
 
LVL 10

Expert Comment

by:makerp
Comment Utility
what are you compiling it with?
0
 

Author Comment

by:Frost_Byte
Comment Utility
VC++ ver 6. Everything is pretty much default in the settings.
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 10

Expert Comment

by:makerp
Comment Utility
strange... can you reproduce this behaviour with a simple console application that just loops and allocates memory..
0
 
LVL 10

Expert Comment

by:oleber
Comment Utility
Can't you work arround this problem.
Cann't you send some information for hard-disk?

If nebeker is correct, did you see how many Virtual Memory you are using?


0
 
LVL 5

Expert Comment

by:nebeker
Comment Utility
Just by chance, are you using a signed integer to hold the amount of memory being requested?  Depending on what type of variable (int, short int, etc.) you're using, and how much memory you're asking for, you might have overflowed the capacity of that variable -- which would mean you're asking for a "negative" amount of memory :)
0
 

Author Comment

by:Frost_Byte
Comment Utility
nebeker : interesting idea, but no. I allocate the same amount of memory twice (this happens when I make a copy of a large _existing_ array). The first time it works, only fails on the second try. That and I calloc an array of length smaller than 10mil of large structs. The command is ...calloc(N * sizeof(BigStruct)). No signed\unsigned issues, as far as I can tell.

oleber : Have around 5gig swap file. I think that should be enough for my needs.

makerp : I'll have to try that but it'll take me time. Will get back to you on that today-tomorrow.

I understand that there is no general known issue with memory limitations, right? There is no hidden switch labeled "let me use everything you got" that I haven't been pushing?
0
 

Expert Comment

by:weicco
Comment Utility
Have tried to call GetLastError / FormatMessage after calloc returns NULL, what does it say?

Hmm... I had wild idea. Try to allocate memory with CreateFileMapping.
0
 
LVL 2

Accepted Solution

by:
jonnin earned 50 total points
Comment Utility
Win NT family suffers (or did) sometimes from memory fragmentation. I.E you have 1 gig of free ram, but (for example) its in alternating pages so you can only allocate 64k or something at a time. This is dated, I don't know if its still an issue, anyone?

Make sure the program does not somewhere allocate and deallocate a little memory thousands of times, which can cause this issue. Alternately, download and run a freeware program called rambooster, I think this may help.  The best fix is not to use dynamic memory anywhere you don't have to (won't fit on the stack, or other reasons like run time determined size, etc).  

If you truely do run out of memory, the "beyone dual processor" versions of windows (2k and/or xp?) can have more than 2gb memory, somewhat more expensive).

0
 
LVL 1

Expert Comment

by:CSLIU
Comment Utility
From 386 to IA32, processors are 32-bit.
The maximum physical addressing space is 4GB in 386 protected mode.

http://support.microsoft.com/support/kb/articles/Q268/3/63.asp

http://support.microsoft.com/support/kb/articles/Q283/0/37.ASP


C code:
/* an example */
#include <stdlib.h>
#include <stdio.h>
int main()
{
 void  *largeP;
 unsigned int uSize;   /* unsigned int is 4bytes(32-bit) in VC++6.0 */

 uSize = 0x80000000 - 1;   /* 2G-1 */

 largeP = malloc( uSize );
 if( largeP == NULL )
 {
   printf("Ok");
   free( largeP );
 }
 else
  printf("Falied");

 return 0;
}
0
 

Expert Comment

by:weicco
Comment Utility
"The best fix is not to use dynamic memory anywhere you don't have to (won't fit on the stack, or other reasons like run time determined size, etc)."

Doesn't malloc use heap not stack?
0
 

Author Comment

by:Frost_Byte
Comment Utility
Well I downloaded some mem-checking utils, not rambooster, but FreeRam XP pro (also free). I still haven't solved the problem but I now know that I have a very fragmented memory which I guess is causing all this. I am, in fact, doing tons of allocating, freeing of small mem fragments. Looks like I have no way of avoiding actual recoding.

Thanks all,
Frost
0
 

Expert Comment

by:weicco
Comment Utility
Little comment... If you need to allocate & realloca & free lots of small memory fragments I suggest you write your own memory manager.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Suggested Solutions

Preface I don't like visual development tools that are supposed to write a program for me. Even if it is Xcode and I can use Interface Builder. Yes, it is a perfect tool and has helped me a lot, mainly, in the beginning, when my programs were small…
This tutorial is posted by Aaron Wojnowski, administrator at SDKExpert.net.  To view more iPhone tutorials, visit www.sdkexpert.net. This is a very simple tutorial on finding the user's current location easily. In this tutorial, you will learn ho…
The goal of this video is to provide viewers with basic examples to understand and use pointers in the C programming language.
The goal of this video is to provide viewers with basic examples to understand how to use strings and some functions related to them in the C programming language.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now