Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

linux process more than 3GB memory

Posted on 2007-08-08
26
Medium Priority
?
449 Views
Last Modified: 2008-01-09
I need to have more than 3GB usable memory in single process. I know i can
- go to 64bit
- use 4G/4G kernel(but then limit is on 4G)
- create large shared memory block, then access it with small window(mmap/munmap). Not whole memory is available at same time, but it's ok for me.

OK, I want to go the last solution. I heard that there some standard/library/kernel-extension that can make this implementation easier for programmer.
I tried to google it out - no results. Anyone knows any references?
0
Comment
Question by:ravenpl
  • 12
  • 9
  • 4
  • +1
26 Comments
 
LVL 46

Accepted Solution

by:
Kent Olsen earned 600 total points
ID: 19653371
Hi raven,

There is a fair amount of flexibility in the implementation of mmap().  Some features are clearly defined in the RFC, others are optional, and remainder can be implemented in any fashion that the O/S supports.

The bottom line is that what works on one 32-bit system may not work on another.

Start by trying to mmap() the entire region.  The length parameter to mmap() is a size_t, normally a 32-bit unsigned int.  Try allocating a single block of 4GB.  If this works, you're set.  :)

But it may not work.  :(  Search for a workable size by halving the length parameter until the call to mmap() succeeds.  Be sure to start with a length of 0x80000000 (2GB).  This will ensure that when you find a usable size that the length of the region is a multiple of the page size.  It is very important that the region be a multiple of a page size.

You can easily swap regions with successive calls to munmap() and mmap().  Keep these to a minimum as the overhead assocated with unmapping/mapping a file/region is considerably more than just a page fault.

  char *Buffer = NULL
  size_t  Length;
  size_t  Start;
 
//  Find an available mapping size.  (Set prot, flags, and fildes as needed)
  Length = 0x80000000;
  Buffer = NULL;
  while (Length)
  {
    Buffer = (char*)mmap(NULL, Length, int prot, int flags, int fildes, NULL);
    if (Buffer)
      break;
    Length /= 2;
  }

To change regions, a small function makes it pretty easy.

void*  Newmmap (size_t NewStart)
{
  if (Buffer)
  {
    munmap (Buffer, Length);
    return (mmap(NULL, Length, int prot, int flags, int fildes, NewStart));
  }
  return (NULL);
}


Good Luck,
Kent
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 19653434
Hi,

First of all you can't go with a traditional 32-bit kernel. With a 32-bit kernell all you have is a 4GB address space and ind it is allocated as 3 GB for all processes and 1 GB kernel space. If you want to go beyond this you'll need one of these hugemem kernels which can address up to 16 GB with a 4 GB per process limit. Another way jou can shose to find some highmem kernels but they are modified by some people and d not a standard like a hugemem kernel. I'd suggest you to go for hugemem kernel to begin with.
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19653448
Yes, I know I can do it (I described in my Q)
But I wanted something like AWE API from Windows.
And sybase is mentioning it...  http://www.sybase.com/detail?id=1030009
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 43

Author Comment

by:ravenpl
ID: 19653617
My previous answer was for Kdo.
KeremE: hugemem kernels can use 64G of memory.
BTW: anybody have some link to performance comparison of standard 3/1G kernel to 4/4G kernel?
0
 
LVL 46

Expert Comment

by:Kent Olsen
ID: 19653690

Hi Raven,

Let's back up and identify your environment.  Can you tell us what hardware / OS is involved here?


Kent
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19653726
32bit kernel, 32bit application(multithreaded), arch x86.
Kdo: I haven't examided Your code, but even it's buggy I know it can be done this way.
0
 
LVL 46

Expert Comment

by:Kent Olsen
ID: 19653749

Which distro?
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19653871
RHEL. I'm aware of hugemem kernel.
0
 
LVL 30

Assisted Solution

by:Kerem ERSOY
Kerem ERSOY earned 1000 total points
ID: 19653908
Ravenpl: Sorry my bad 16GB is generally th lower limit though some applications suggest that. But anyway you'll need to use a hugemem kernel for accomotading a process larger than 3 GB. In a traditional 1G/3G 32 bit system you'll have only 2.4 to 2.7 GB free memory. furthermote swap window on these sytem is 128 MB so it wil take lots of time swapping your porces in /out.

In a hugemem system you'll have a 4GB process space with up to 64 GB memory addressing with a swap windows of 1GB (insimapped into 4G / 4G split memory's kernel memory block)

this is the only way you can go. Other than that you can go to create a shm file. Which can be as large as 64 GB on a hugekernel. (this is what SyBase does). In linux you don't need AWE since a process can address up to 4 GB memory by itself.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 19653958
Or you can use SMP kerne_ which can address as much as 16 GB. with 4 GB process size like Hugemem kernel and user space drivers won't need to be hugemem special ones. You can use standard applications with SMP kernel . Furthermore RHEL does not support hugemem in RHEL5 and you need to upgrade your support contract to RHEL AS support for RedHAT ES support will only cover your system if it has 16 GB or less memory.
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19654021
Wait. Please read my Q again. I'm aware of 4G/4G(hugemem) and 64bit systems. For some reasons I don;t want port the application to 64bit, and 4G is really temporary solution(and possibly have severe performance impact).
0
 
LVL 46

Expert Comment

by:Kent Olsen
ID: 19654056

Hi Raven,

We're kind of wandering around in the dark now.  You can do your own mapping in a 1G/3G system, install a kernel that supports 4G or greater, or upgrade the to 64 bits (which seems to be your least preferred solution).

What more can we suggest?

Kent
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 19654108
Raven: You have 3 alternatives (except from porting your app to 64 bit)
- Hugemem: Supports up to 64 GB, there's a need for compileing some modules. No need to modify your code. Why do you think is a temporary solution? What makes it temporary ?
- SMP: The moust useful no need extra compiled modules. No nned to modify your code.
- Highmem : This is the one with the greatest performance impact since kernel boundaries are at limit pthere's a process penalty of nearly 6%

These are only alternatives. Other than that with a standard kernel yu can not go beyong this bloody 3GB limit.
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19654158
> What more can we suggest?
A library, API, or kernel support for the mapping. That was the question. Really.

KeremE:
- Hugemem: 64G for process? Not at all. Please read more.
- SMP? can You elaborate how SMP can give me more ram for process?
- Highmem? Again, can You elaborate how HM can give me more ram for process?
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19654372
BTW: there's even easier way than mmap/munmap: shmat/shmdt with many shm areas.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 19654738
Raven:
Hugemem: I told "Supports up to 64 GB" of memory, of course. 4 GB per process memory (as we'd talked earlier). Your program will work with work without modification and it will be able to allocate more than 3 GB.
SMP: 4 GB per process / 16 GB max memory. Your program will work with work without modification and it will be able to allocate more than 3 GB.
Highmem: Uses PAE up to 64 GB of mem / 4gb per process but with penalty 6% slower.

In these solution you won't need any extra API or something since all memory addressing thing is addressed by the kernel. All you'd do is to call vmalloc/ioremap to alloate more tha 3GB and it will succeed automatically if you use one of these kernels. They are only around 5% slower than the normal counterpart. But this is quite tolarable.

Another alternative isto use 64 architecture. it has a 4GB per process limitation for x32 applications but 512GB !!! per process limit usable with vmalloc/ioremap for native x86_64 application.
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 19654756
BTW - As KDO told earlier what other alternatives you're expect? Using a different kernel (SMP for ex.) requiring no change to your code ins't simple enough?
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19654798
> Another alternative isto use 64 architecture. it has a 4GB per process limitation for x32 applications but 512GB !!!
Oh, it's much more than that - believe me, I was testing ;)
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19654807
I already pointed out what I expect.
0
 
LVL 30

Assisted Solution

by:Kerem ERSOY
Kerem ERSOY earned 1000 total points
ID: 19654917
yeah but even the source you've quoted says there's no such a think. They do what I earlier told you. They create a /dev/shm file which can be as much as 64 GB and use this file.

You wbnt something that is not exist :)
0
 
LVL 1

Assisted Solution

by:ostwaldl
ostwaldl earned 400 total points
ID: 19657000
ravenpl - No API call is going to solve the problem the way you want...its an architectural limitation and you have to switch to a "hugemem" kernel or port to a 64 bit architecture to accomplish this.  A good primer on memory management on 32 bit Linux systems:  

http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html

0
 
LVL 43

Author Comment

by:ravenpl
ID: 19657116
Of course the target solution is 64bit. But before I have to workaround it. And if You read the thread, You should notice that I know how to accomplish that.
My question was if maybe there's some library ready for that. Or maybe there's some kernel patch which helps with this task. As You noticed Windows has such thing.
0
 
LVL 30

Assisted Solution

by:Kerem ERSOY
Kerem ERSOY earned 1000 total points
ID: 19658273
Lets approach it from the other end. What AWE does is to allow processes to address up to 64 GB memory space. Why does Windows (NT4 and Windows 2000 but not windows 2003) needs that ? Because they both operate in the traditomal 4GB memory model and they don't have an option like having a modified kernel which will handle it for them. L1nux side is quite different. Linnux is capable of addressing the memory issue all by modifying its kernel. So someth_n like AWE and API is quite useless for linux.

RavenPL: will you please download the .dpf document that you've suggested for SyBase. It explains how SyBASE is able to use 64 GB memory on Linux (of course on a hugemem kernel)
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19659987
OK, no such thing under linux (like AWE). Pitty. Not what I expected though.
0
 
LVL 43

Author Comment

by:ravenpl
ID: 19659996
I'm going to make performance measurement: /dev/shm/+mmap against shmat/shmdt.
That's what left to me, right?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 19660221
Since it is a VM file I don't think the performance will be very bad unless you do so much of swappng due to lack of memory. Most sources say unser nowmal conditions the performance decrease will be around  4% to 6%. You'll need lots of memory considering even the descriptors and pointers required to handle this file will be around 200 MB :)

As for this is not what you expect :) as EE experts we know that "ok you can't do it" counts as a valid answer in EE :)

Good luck
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
Suggested Courses
Course of the Month10 days, 15 hours left to enroll

571 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question