linux process more than 3GB memory

I need to have more than 3GB usable memory in single process. I know i can
- go to 64bit
- use 4G/4G kernel(but then limit is on 4G)
- create large shared memory block, then access it with small window(mmap/munmap). Not whole memory is available at same time, but it's ok for me.

OK, I want to go the last solution. I heard that there some standard/library/kernel-extension that can make this implementation easier for programmer.
I tried to google it out - no results. Anyone knows any references?
LVL 43
ravenplAsked:
Who is Participating?
 
Kent OlsenData Warehouse Architect / DBACommented:
Hi raven,

There is a fair amount of flexibility in the implementation of mmap().  Some features are clearly defined in the RFC, others are optional, and remainder can be implemented in any fashion that the O/S supports.

The bottom line is that what works on one 32-bit system may not work on another.

Start by trying to mmap() the entire region.  The length parameter to mmap() is a size_t, normally a 32-bit unsigned int.  Try allocating a single block of 4GB.  If this works, you're set.  :)

But it may not work.  :(  Search for a workable size by halving the length parameter until the call to mmap() succeeds.  Be sure to start with a length of 0x80000000 (2GB).  This will ensure that when you find a usable size that the length of the region is a multiple of the page size.  It is very important that the region be a multiple of a page size.

You can easily swap regions with successive calls to munmap() and mmap().  Keep these to a minimum as the overhead assocated with unmapping/mapping a file/region is considerably more than just a page fault.

  char *Buffer = NULL
  size_t  Length;
  size_t  Start;
 
//  Find an available mapping size.  (Set prot, flags, and fildes as needed)
  Length = 0x80000000;
  Buffer = NULL;
  while (Length)
  {
    Buffer = (char*)mmap(NULL, Length, int prot, int flags, int fildes, NULL);
    if (Buffer)
      break;
    Length /= 2;
  }

To change regions, a small function makes it pretty easy.

void*  Newmmap (size_t NewStart)
{
  if (Buffer)
  {
    munmap (Buffer, Length);
    return (mmap(NULL, Length, int prot, int flags, int fildes, NewStart));
  }
  return (NULL);
}


Good Luck,
Kent
0
 
Kerem ERSOYPresidentCommented:
Hi,

First of all you can't go with a traditional 32-bit kernel. With a 32-bit kernell all you have is a 4GB address space and ind it is allocated as 3 GB for all processes and 1 GB kernel space. If you want to go beyond this you'll need one of these hugemem kernels which can address up to 16 GB with a 4 GB per process limit. Another way jou can shose to find some highmem kernels but they are modified by some people and d not a standard like a hugemem kernel. I'd suggest you to go for hugemem kernel to begin with.
0
 
ravenplAuthor Commented:
Yes, I know I can do it (I described in my Q)
But I wanted something like AWE API from Windows.
And sybase is mentioning it...  http://www.sybase.com/detail?id=1030009
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
ravenplAuthor Commented:
My previous answer was for Kdo.
KeremE: hugemem kernels can use 64G of memory.
BTW: anybody have some link to performance comparison of standard 3/1G kernel to 4/4G kernel?
0
 
Kent OlsenData Warehouse Architect / DBACommented:

Hi Raven,

Let's back up and identify your environment.  Can you tell us what hardware / OS is involved here?


Kent
0
 
ravenplAuthor Commented:
32bit kernel, 32bit application(multithreaded), arch x86.
Kdo: I haven't examided Your code, but even it's buggy I know it can be done this way.
0
 
Kent OlsenData Warehouse Architect / DBACommented:

Which distro?
0
 
ravenplAuthor Commented:
RHEL. I'm aware of hugemem kernel.
0
 
Kerem ERSOYPresidentCommented:
Ravenpl: Sorry my bad 16GB is generally th lower limit though some applications suggest that. But anyway you'll need to use a hugemem kernel for accomotading a process larger than 3 GB. In a traditional 1G/3G 32 bit system you'll have only 2.4 to 2.7 GB free memory. furthermote swap window on these sytem is 128 MB so it wil take lots of time swapping your porces in /out.

In a hugemem system you'll have a 4GB process space with up to 64 GB memory addressing with a swap windows of 1GB (insimapped into 4G / 4G split memory's kernel memory block)

this is the only way you can go. Other than that you can go to create a shm file. Which can be as large as 64 GB on a hugekernel. (this is what SyBase does). In linux you don't need AWE since a process can address up to 4 GB memory by itself.
0
 
Kerem ERSOYPresidentCommented:
Or you can use SMP kerne_ which can address as much as 16 GB. with 4 GB process size like Hugemem kernel and user space drivers won't need to be hugemem special ones. You can use standard applications with SMP kernel . Furthermore RHEL does not support hugemem in RHEL5 and you need to upgrade your support contract to RHEL AS support for RedHAT ES support will only cover your system if it has 16 GB or less memory.
0
 
ravenplAuthor Commented:
Wait. Please read my Q again. I'm aware of 4G/4G(hugemem) and 64bit systems. For some reasons I don;t want port the application to 64bit, and 4G is really temporary solution(and possibly have severe performance impact).
0
 
Kent OlsenData Warehouse Architect / DBACommented:

Hi Raven,

We're kind of wandering around in the dark now.  You can do your own mapping in a 1G/3G system, install a kernel that supports 4G or greater, or upgrade the to 64 bits (which seems to be your least preferred solution).

What more can we suggest?

Kent
0
 
Kerem ERSOYPresidentCommented:
Raven: You have 3 alternatives (except from porting your app to 64 bit)
- Hugemem: Supports up to 64 GB, there's a need for compileing some modules. No need to modify your code. Why do you think is a temporary solution? What makes it temporary ?
- SMP: The moust useful no need extra compiled modules. No nned to modify your code.
- Highmem : This is the one with the greatest performance impact since kernel boundaries are at limit pthere's a process penalty of nearly 6%

These are only alternatives. Other than that with a standard kernel yu can not go beyong this bloody 3GB limit.
0
 
ravenplAuthor Commented:
> What more can we suggest?
A library, API, or kernel support for the mapping. That was the question. Really.

KeremE:
- Hugemem: 64G for process? Not at all. Please read more.
- SMP? can You elaborate how SMP can give me more ram for process?
- Highmem? Again, can You elaborate how HM can give me more ram for process?
0
 
ravenplAuthor Commented:
BTW: there's even easier way than mmap/munmap: shmat/shmdt with many shm areas.
0
 
Kerem ERSOYPresidentCommented:
Raven:
Hugemem: I told "Supports up to 64 GB" of memory, of course. 4 GB per process memory (as we'd talked earlier). Your program will work with work without modification and it will be able to allocate more than 3 GB.
SMP: 4 GB per process / 16 GB max memory. Your program will work with work without modification and it will be able to allocate more than 3 GB.
Highmem: Uses PAE up to 64 GB of mem / 4gb per process but with penalty 6% slower.

In these solution you won't need any extra API or something since all memory addressing thing is addressed by the kernel. All you'd do is to call vmalloc/ioremap to alloate more tha 3GB and it will succeed automatically if you use one of these kernels. They are only around 5% slower than the normal counterpart. But this is quite tolarable.

Another alternative isto use 64 architecture. it has a 4GB per process limitation for x32 applications but 512GB !!! per process limit usable with vmalloc/ioremap for native x86_64 application.
0
 
Kerem ERSOYPresidentCommented:
BTW - As KDO told earlier what other alternatives you're expect? Using a different kernel (SMP for ex.) requiring no change to your code ins't simple enough?
0
 
ravenplAuthor Commented:
> Another alternative isto use 64 architecture. it has a 4GB per process limitation for x32 applications but 512GB !!!
Oh, it's much more than that - believe me, I was testing ;)
0
 
ravenplAuthor Commented:
I already pointed out what I expect.
0
 
Kerem ERSOYPresidentCommented:
yeah but even the source you've quoted says there's no such a think. They do what I earlier told you. They create a /dev/shm file which can be as much as 64 GB and use this file.

You wbnt something that is not exist :)
0
 
ostwaldlCommented:
ravenpl - No API call is going to solve the problem the way you want...its an architectural limitation and you have to switch to a "hugemem" kernel or port to a 64 bit architecture to accomplish this.  A good primer on memory management on 32 bit Linux systems:  

http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html

0
 
ravenplAuthor Commented:
Of course the target solution is 64bit. But before I have to workaround it. And if You read the thread, You should notice that I know how to accomplish that.
My question was if maybe there's some library ready for that. Or maybe there's some kernel patch which helps with this task. As You noticed Windows has such thing.
0
 
Kerem ERSOYPresidentCommented:
Lets approach it from the other end. What AWE does is to allow processes to address up to 64 GB memory space. Why does Windows (NT4 and Windows 2000 but not windows 2003) needs that ? Because they both operate in the traditomal 4GB memory model and they don't have an option like having a modified kernel which will handle it for them. L1nux side is quite different. Linnux is capable of addressing the memory issue all by modifying its kernel. So someth_n like AWE and API is quite useless for linux.

RavenPL: will you please download the .dpf document that you've suggested for SyBase. It explains how SyBASE is able to use 64 GB memory on Linux (of course on a hugemem kernel)
0
 
ravenplAuthor Commented:
OK, no such thing under linux (like AWE). Pitty. Not what I expected though.
0
 
ravenplAuthor Commented:
I'm going to make performance measurement: /dev/shm/+mmap against shmat/shmdt.
That's what left to me, right?
0
 
Kerem ERSOYPresidentCommented:
Since it is a VM file I don't think the performance will be very bad unless you do so much of swappng due to lack of memory. Most sources say unser nowmal conditions the performance decrease will be around  4% to 6%. You'll need lots of memory considering even the descriptors and pointers required to handle this file will be around 200 MB :)

As for this is not what you expect :) as EE experts we know that "ok you can't do it" counts as a valid answer in EE :)

Good luck
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.