Overcoming 32MB limit boundary for malloc on Linux


Using gcc on RHEL 5.1, I've allocated some memory using malloc (134,217,728) and malloc returns the pointer to the memory successfully, but I always receive a segmentation fault after trying to index after 32MB, I assume there is a 32MB boundary in place.

Is there anyway to increase this limit?

Many thanks.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

evilrixSenior Software Engineer (Avast)Commented:
You should be able to allocate and address up to 2GB of memory. Are you sure this is the problem? Have you tried creating a small dummy program to test this?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
That should work just fine so your segmentation fault has nothing to do with a 32MB boundary. I ran the program below, and it runs just fine, and I even allocated 2,147,483,647 with no problems (I have 4GB RAM).

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
int main(int argc, char **argv)
   unsigned long iX;
   size_t bufsize = 134217728;
   char *pBuf = malloc(bufsize);
   if (!pBuf)
      printf("Allocation failed\n");
      return -1;
   for (iX=0; iX<bufsize; iX++)
     if (iX % 0x1000 == 0)
         printf("0x%08X\n", iX);
      pBuf[iX] = 0xFF;
      pBuf[iX] = 0x00;
   printf("Filled to 0x%04X\n", iX);

Open in new window

evilrixSenior Software Engineer (Avast)Commented:
Try the following code to see it if fails. If it doesn't the problem lays somewhere else.

#include <malloc.h>
#include <assert.h>
int main()
        char * p = malloc(0xFFFFFFF);
        return 0;

Open in new window

Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

evilrixSenior Software Engineer (Avast)Commented:
BTW: I know I forgot the call to free() -- but it's only to test so I'm not too concerned about a memory leak :)
can you show us the code where you allocate the memory and where you access it?

any chance you dont access it through a byte-pointer (unsigned char or char), like:

struct any_big_object { ... };

any_big_object *pobjs = (any_big_object *)malloc(134.217.728);

// this accesses the byte at 32MB (33554432), you have to cast pobjs to char-pointer before adding the offset for it
char i = *((char*)pobjs + 33554432);
// or
char i = ((char*)pobjs)[335544329];

// this accesses the struct at position 335544329, which might fail, because it accesses the byte 335544329*sizeof(any_big_object)
char i = *(char*)(pobjs + 33554432);
char i = ((char*)pobjs)[33554432];

well, it might be easier if we see your code .. ;)

evilrixSenior Software Engineer (Avast)Commented:
Interesting article you might find useful.

Brent-CampbellAuthor Commented:
Many thanks for all of your comments.  I've actually discovered this isn't the problem, the problem was actually due to jumping four bytes rather than one in a for loop and therefore accessing unallocated memory!
evilrixSenior Software Engineer (Avast)Commented:
Like I said, "Are you sure this is the problem?" :) http:#20778613
Technically, the 2GB limit is correct, yet some other runtime constraints may lower that limit. Take a look at http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html ("When Linux Runs Out of Memory") for a more in-depth view.

BTW, to learn more about system limts, check out 'getrlimit()' (http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html)
evilrixSenior Software Engineer (Avast)Commented:
Hi Brent-Campbell,

Did you need anymore assistance with this?

Brent-CampbellAuthor Commented:
Sorry for the huge delay, thanks for your time and efforts and realising that the problem was elsewhere!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux OS Dev

From novice to tech pro — start learning today.