Overcoming 32MB limit boundary for malloc on Linux

That should work just fine so your segmentation fault has nothing to do with a 32MB boundary. I ran the program below, and it runs just fine, and I even allocated 2,147,483,647 with no problems (I have 4GB RAM).

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
 
int main(int argc, char **argv)
{
   unsigned long iX;
   size_t bufsize = 134217728;
   char *pBuf = malloc(bufsize);
 
   if (!pBuf)
   {
      printf("Allocation failed\n");
      return -1;
   }
 
   for (iX=0; iX<bufsize; iX++)
   {
     if (iX % 0x1000 == 0)
         printf("0x%08X\n", iX);
      pBuf[iX] = 0xFF;
      pBuf[iX] = 0x00;
   }
   free(pBuf);
   printf("Filled to 0x%04X\n", iX);
   return(0);
}

Select allOpen in new window

Try the following code to see it if fails. If it doesn't the problem lays somewhere else.

#include <malloc.h>
#include <assert.h>
 
int main()
{
        char * p = malloc(0xFFFFFFF);
        assert(p);
        p[0xFFFFFFE]=0;
 
        return 0;
}

Select allOpen in new window

BTW: I know I forgot the call to free() -- but it's only to test so I'm not too concerned about a memory leak :)

can you show us the code where you allocate the memory and where you access it?

any chance you dont access it through a byte-pointer (unsigned char or char), like:

struct any_big_object { ... };

any_big_object *pobjs = (any_big_object *)malloc(134.217.728);

// this accesses the byte at 32MB (33554432), you have to cast pobjs to char-pointer before adding the offset for it
char i = *((char*)pobjs + 33554432);
// or
char i = ((char*)pobjs)[335544329];

// this accesses the struct at position 335544329, which might fail, because it accesses the byte 335544329*sizeof(any_big_object)
char i = *(char*)(pobjs + 33554432);
or
char i = ((char*)pobjs)[33554432];


well, it might be easier if we see your code .. ;)

ike

Interesting article you might find useful.

http://www.linuxjournal.com/article/6390

Many thanks for all of your comments.  I've actually discovered this isn't the problem, the problem was actually due to jumping four bytes rather than one in a for loop and therefore accessing unallocated memory!

Technically, the 2GB limit is correct, yet some other runtime constraints may lower that limit. Take a look at http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html ("When Linux Runs Out of Memory") for a more in-depth view.

BTW, to learn more about system limts, check out 'getrlimit()' (http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html)

Hi Brent-Campbell,

Did you need anymore assistance with this?

-Rx

Sorry for the huge delay, thanks for your time and efforts and realising that the problem was elsewhere!