Overcoming 32MB limit boundary for malloc on Linux


Using gcc on RHEL 5.1, I've allocated some memory using malloc (134,217,728) and malloc returns the pointer to the memory successfully, but I always receive a segmentation fault after trying to index after 32MB, I assume there is a 32MB boundary in place.

Is there anyway to increase this limit?

Many thanks.
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

evilrixConnect With a Mentor Senior Software Engineer (Avast)Commented:
You should be able to allocate and address up to 2GB of memory. Are you sure this is the problem? Have you tried creating a small dummy program to test this?
That should work just fine so your segmentation fault has nothing to do with a 32MB boundary. I ran the program below, and it runs just fine, and I even allocated 2,147,483,647 with no problems (I have 4GB RAM).

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
int main(int argc, char **argv)
   unsigned long iX;
   size_t bufsize = 134217728;
   char *pBuf = malloc(bufsize);
   if (!pBuf)
      printf("Allocation failed\n");
      return -1;
   for (iX=0; iX<bufsize; iX++)
     if (iX % 0x1000 == 0)
         printf("0x%08X\n", iX);
      pBuf[iX] = 0xFF;
      pBuf[iX] = 0x00;
   printf("Filled to 0x%04X\n", iX);

Open in new window

evilrixSenior Software Engineer (Avast)Commented:
Try the following code to see it if fails. If it doesn't the problem lays somewhere else.

#include <malloc.h>
#include <assert.h>
int main()
        char * p = malloc(0xFFFFFFF);
        return 0;

Open in new window

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

evilrixSenior Software Engineer (Avast)Commented:
BTW: I know I forgot the call to free() -- but it's only to test so I'm not too concerned about a memory leak :)
can you show us the code where you allocate the memory and where you access it?

any chance you dont access it through a byte-pointer (unsigned char or char), like:

struct any_big_object { ... };

any_big_object *pobjs = (any_big_object *)malloc(134.217.728);

// this accesses the byte at 32MB (33554432), you have to cast pobjs to char-pointer before adding the offset for it
char i = *((char*)pobjs + 33554432);
// or
char i = ((char*)pobjs)[335544329];

// this accesses the struct at position 335544329, which might fail, because it accesses the byte 335544329*sizeof(any_big_object)
char i = *(char*)(pobjs + 33554432);
char i = ((char*)pobjs)[33554432];

well, it might be easier if we see your code .. ;)

evilrixSenior Software Engineer (Avast)Commented:
Interesting article you might find useful.

Brent-CampbellAuthor Commented:
Many thanks for all of your comments.  I've actually discovered this isn't the problem, the problem was actually due to jumping four bytes rather than one in a for loop and therefore accessing unallocated memory!
evilrixConnect With a Mentor Senior Software Engineer (Avast)Commented:
Like I said, "Are you sure this is the problem?" :) http:#20778613
Technically, the 2GB limit is correct, yet some other runtime constraints may lower that limit. Take a look at http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html ("When Linux Runs Out of Memory") for a more in-depth view.

BTW, to learn more about system limts, check out 'getrlimit()' (http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html)
evilrixSenior Software Engineer (Avast)Commented:
Hi Brent-Campbell,

Did you need anymore assistance with this?

Brent-CampbellAuthor Commented:
Sorry for the huge delay, thanks for your time and efforts and realising that the problem was elsewhere!
All Courses

From novice to tech pro — start learning today.