Solved

1.2 GB Memory Limit Per .NET Process

Posted on 2004-08-14
31
2,119 Views
Last Modified: 2008-01-09
How do you get rid of the 1.2 GB Memory Limit Per Process in .NET?

This problem is described extensively (by Jeff) in the following topic:

http://www.error-bank.com/microsoft.public.dotnet.framework.aspnet.webservices/23271_Thread.aspx

Essentially, a single .NET executable written in C# (in my case) can only allocate about 1.1-1.2 GB of memory before throwing an Out of Memory exception. This is completely independent of how much actual memory a machine has. I tested it on a machine with 2GB Ram, and on a dual Xeon with 4GB Ram. Same results. In both cases an executable can only obtain 1.2 GB from the system before crashing.

The /3GB switch in boot.ini does nothing to alleviate or fix the problem. I should probably also point out that SEVERAL identical .NET processes together easily use up to 3.8 GB of RAM. It's just that any single .NET process is somehow capped at using only 1.2 GB.

Does anyone know what exactly is causing this problem?
0
Comment
Question by:IH666
  • 7
  • 6
  • 5
  • +4
31 Comments
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804385
I have also run into these issues .... I have pushed it up to <> 2 gb using the method mentioned in that thread. Out of curiosity are you running as a normal app or as a web app ? Another thing I will bring up as a possible work around is running in multiple process spaces because as you say above you can run multiple processes with 1 gb ram.
0
 

Author Comment

by:IH666
ID: 11804532
The architecture of the program prevents it from running in multiple instances (believe me, I looked into that extensively). My only hope is in fixing this problem. The 2GB you mention, that does not apply .NET, does it?

The program is just a regular executable.
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804598
yeah using the methods described in that article I got to around 2 gb before running into the issue (large cache) ...
0
 

Author Comment

by:IH666
ID: 11804608
Which method specifically, the /3GB switch?
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804620
and rebasing .dlls
0
 

Author Comment

by:IH666
ID: 11804693
That link mentioned there is dead unfortunately. Can you explain what rebasing DLLs is?
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804702
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 11807067

A bit off topic, but here is something to keep in mind...


Windows 2000 Server Edition has an upper limit of 2 Gig Ram allocation for any single application.  You will need to use Windows Advanced, or  Windows DataCenter in order to allocate more than 2 Gig of RAM for any single application.




0
 

Author Comment

by:IH666
ID: 11815181
This is using Win 2003 Std.
0
 

Author Comment

by:IH666
ID: 11816396
So, there's no info on what exactly is causing the problem and whether this would be fixed in later releases of .NET?
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11818319
This article has a VERY good explanation of what is going on:
http://www.brianmadden.com/content/content.asp?id=69

-Eric
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11818339
Also, that article mentions that the bootswitch equivilant to the windows 2000 /3GB is, for windows 2003, /4GT.  This should give you a bit more to play with, so long as you are not running in a terminal services environment.

-Eric
0
 

Author Comment

by:IH666
ID: 11819024
I read through this, and as I understood from the comments, the only solution is to upgrade to 64-bit Win?

I presume any version (including Standard) would work, on any type of processor.

Right?
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 11819345


to my knowledge, .net is not supported in 64-bit Win
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 11819960
WOWing Your 32-Bit Applications with 64-Bit Windows Part 1, March 8, 2004
http://www.devx.com/amd/Article/20342

>>"Microsoft hasn't yet ported the .NET Framework to 64-bit Windows, and the 32-bit version will not run on that environment—so, you can't run any .NET apps on 64-bit Windows"

Bob
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 3

Accepted Solution

by:
bigjim2000 earned 500 total points
ID: 11821459
It looks to me like you need to get a 64-Bit computer, get 64-Bit windows, and wait a few more months for the release of 64-bit .NET framework before you can continue.  Until then, I'd suggest trying to get by with the /4GT bootswitch.

-Eric
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11821590
Also, have you considered using a process pool full of applications designed specifically to add more available memory?

If you need just the memory, and don't need it to be incredibly efficient, then you might stand a chance going that route.

It could be set up relatively easy with remoting (but it would suffer more of an overhead cost than using, say, a named pipe).  It would be even easier if each process simply held an array of a certain data-type.  At one point I was working on a project that needed an array of roughly 115 billion booleans (about 3GB).  I ran this in 3 processes, and had each one hold about 38Billion.  Of course, the apps did extra proccessing internally, so that I didn't have to index each one individually....it was a mess.... and still slow.... but it might be a route for you.

-Eric
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11821683
out of curiosity ... with this size array why not just store it in file and map views of the file directly into your memory space ?
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 11821687
Some heavy reading from Raymond Chen:  

Why is the virtual address space 4GB anyway?
http://weblogs.asp.net/oldnewthing/archive/2004/08/17/215682.aspx

Myth: The /3GB switch lets me map one giant 3GB block of memory
http://weblogs.asp.net/oldnewthing/archive/2004/08/16/215089.aspx

Myth: The /3GB switch expands the user-mode address space of all programs
http://weblogs.asp.net/oldnewthing/archive/2004/08/12/213468.aspx

Myth: You need /3GB if you have more than 2GB of physical memory
http://weblogs.asp.net/oldnewthing/archive/2004/08/11/212720.aspx

Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory
http://weblogs.asp.net/oldnewthing/archive/2004/08/10/211890.aspx

Myth: Without /3GB the total amount of memory that can be allocated across all programs is 2GB
http://weblogs.asp.net/oldnewthing/archive/2004/08/09/211356.aspx

Kernel address space consequences of the /3GB switch
http://weblogs.asp.net/oldnewthing/archive/2004/08/06/209840.aspx

The oft-misunderstood /3GB switch
http://weblogs.asp.net/oldnewthing/archive/2004/08/05/208908.aspx

Bob
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11821737
then have a process pool that operates upon the disk backed shared memory :)
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11821864
gregoryyoung has a good point, heh.

-Eric
0
 

Author Comment

by:IH666
ID: 11856584
/4GB still only lets the app use 1.2GB before crashing. Is that the expected behavior?
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11873476
You can obtain almost all physical memory available in the computer to your app.

Microsoft has a technology that is there for quite a long while that enables using more than 2GB of memory in a single process.
The technology is called Address Windowing Extensions (AWE):
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenlib/html/awewindata.asp

This will not enable .NET allocations of objects, but it enables allocations of unmanaged memory limited only by physical pages.
However, you cannot use all of this memory at the same time.
The basic steps of using AWE memory are:
1. Allocate physical pages associated with your process (using AllocateUserPhysicalPages)
2. Reserve an address space region (not memory) for the physical memory (using VirtualAlloc with MEM_RESERVE | MEM_PHYSICAL)
2. Map the physical pages into that address space region (using MapUserPhysicalPages)
3. <Use that memory>
4. Unmap the physical pages from the address space region (using MapUserPhysicalPages with last parameter of NULL)
5. Release the reserved address space region (using VirtualFree)
6. Free physical pages (using FreeUserPhysicalPages)

This way you can use almost all physical memory in the computer, mapping portions of it to your address space at a time.

Important Note:
To use AWE, you need the "Lock Pages In Memory" privilege. This requires editing of the computer's local security policy (since this is not enabled by default) and some extra code to enable the privilege in your process.
All of this is in the MSDN examples.

*** This technology does not require special operating systems and is supported from Windows 2000 onwards ***

Hope this helps,

Eran
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11874925
I once wrote wrappers for FileMapping which is very close to AWE but doesn't need a special privileg. You can find it here: http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=36988E96-BA51-4FD5-AF73-AD917B50DF1F
You also may use it as base to write a wrapper for AWE.
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11878124
Keep in mind that using FileMapping has two major drawbacks when compared to AWE:

In order to use FileMapping as a paging mechanism, you must have a filesystem object to back the mapped memory (either your own file or the paging file). This means that you either must create a large file, or risk enlarging the system's page file. These operations include some heavy I/O operations and may introduce a considerable amount of time.

Another important drawback is the fact that the backing store must be read into memory before you can write to that memory (except first usage of pagefile backed memory). So if you use your own file, before your code can touch a single byte of mapped memory (actually at the moment it does), the relevant page is read from the file into memory. Since you obviously read a lot, this will create a lot of I/O, not to mention that every writing of bytes into the mapped memory will also write them to the filesystem.

With AWE no implicit I/O operations are performed. You have full control over the memory.
For example, you could hold the persistant data in compressed format and decompress it into AWE memory. with FileMapping, this cannot be achieved...
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11885572
>In order to use FileMapping as a paging mechanism, you must have a filesystem object to back the mapped memory (either your own file or the paging file).
I think this is always the case (with AWE it's always the paging file) - but the bytes are not necessarily written to the file. The paging file should already be large enough so there is no extra overhead since it exists anyway. With both you can only access ~1GB of the memory at once. In case that you have 4GB memory and access 3GB of it there might be a difference in the behavior but to be sure I would have to test it. AWE will probably be better choice if it's important to avoid paging as much as possible. But be aware that you force all other applications paging their memory to disk by allocating a huge part of the physical memory. AWE is very good to access memory from a 32-Bit Process on a machine with very much memory (e.g. a 64-Bit machine with 32GB memory). AWE will help your program to run better on future machines, but it doesn't completely solve the current problem.
Be aware how memory management works under the target operating system. Make tests, then decide what you will use.
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11886427
ptmcomp:
A minor correction on your last comment.
When using AWE, the memory you allocate is a set of LOCKED physical pages (in physical memory).
It is never swapped out and therefore is NOT backed by the pagefile.

If you free the physical pages, the data is simply lost. As long as it is allocated, it is locked in memory and is not swapped out.

With memory mapped files, the physical memory used for the mapped file can be paged out and therefore needs a backing store
(there is no guarantee that the memory is physical)

So basically, AWE does not require enlarging the pagefile or any other filesystem related operations.
It is your responsibility to control the locking, contents, and release of the actual physical memory.

I know all of this because I have used AWE as a mechanism to access large volumes of data in proven projects.
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11886675
Yes, you're right about that. You can lock the physical pages (or they are even always locked). If you have only 4GB memory this means that all other processes share the left over physical memory, so if you allocate more than 80% of the physical memory the system's performance will probably be near zero. AWE is great if you have lots of physical memory. My wrappers for file mapping contains wrappers to access unmanaged memory which you could use with AWE, if you like to. But I didn't find a way to avoid the memory to be copied while accessing it.
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11886921
We prevented the copying using unsafe code and IntPtr.
When we lock a large object, we return an IntPtr that points to its memory.

We were able to simultanously access a little over 2.5GB of memory (32bit machine, 4GB RAM, /3GB flag)
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11887067
If you need maximum memory and performance you will need unsafe code, you're right.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: Najam
Having new technologies does not mean they will completely replace old components.  Recently I had to create WCF that will be called by VB6 component.  Here I will describe what steps one should follow while doing so, please feel free to post any qu…
Introduction This article series is supposed to shed some light on the use of IDisposable and objects that inherit from it. In essence, a more apt title for this article would be: using (IDisposable) {}. I’m just not sure how many people would ge…
Along with being a a promotional video for my three-day Annielytics Dashboard Seminor, this Micro Tutorial is an intro to Google Analytics API data.
Migrating to Microsoft Office 365 is becoming increasingly popular for organizations both large and small. If you have made the leap to Microsoft’s cloud platform, you know that you will need to create a corporate email signature for your Office 365…

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now