Solved

1.2 GB Memory Limit Per .NET Process

Posted on 2004-08-14
31
2,106 Views
Last Modified: 2008-01-09
How do you get rid of the 1.2 GB Memory Limit Per Process in .NET?

This problem is described extensively (by Jeff) in the following topic:

http://www.error-bank.com/microsoft.public.dotnet.framework.aspnet.webservices/23271_Thread.aspx

Essentially, a single .NET executable written in C# (in my case) can only allocate about 1.1-1.2 GB of memory before throwing an Out of Memory exception. This is completely independent of how much actual memory a machine has. I tested it on a machine with 2GB Ram, and on a dual Xeon with 4GB Ram. Same results. In both cases an executable can only obtain 1.2 GB from the system before crashing.

The /3GB switch in boot.ini does nothing to alleviate or fix the problem. I should probably also point out that SEVERAL identical .NET processes together easily use up to 3.8 GB of RAM. It's just that any single .NET process is somehow capped at using only 1.2 GB.

Does anyone know what exactly is causing this problem?
0
Comment
Question by:IH666
  • 7
  • 6
  • 5
  • +4
31 Comments
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804385
I have also run into these issues .... I have pushed it up to <> 2 gb using the method mentioned in that thread. Out of curiosity are you running as a normal app or as a web app ? Another thing I will bring up as a possible work around is running in multiple process spaces because as you say above you can run multiple processes with 1 gb ram.
0
 

Author Comment

by:IH666
ID: 11804532
The architecture of the program prevents it from running in multiple instances (believe me, I looked into that extensively). My only hope is in fixing this problem. The 2GB you mention, that does not apply .NET, does it?

The program is just a regular executable.
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804598
yeah using the methods described in that article I got to around 2 gb before running into the issue (large cache) ...
0
 

Author Comment

by:IH666
ID: 11804608
Which method specifically, the /3GB switch?
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804620
and rebasing .dlls
0
 

Author Comment

by:IH666
ID: 11804693
That link mentioned there is dead unfortunately. Can you explain what rebasing DLLs is?
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11804702
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 11807067

A bit off topic, but here is something to keep in mind...


Windows 2000 Server Edition has an upper limit of 2 Gig Ram allocation for any single application.  You will need to use Windows Advanced, or  Windows DataCenter in order to allocate more than 2 Gig of RAM for any single application.




0
 

Author Comment

by:IH666
ID: 11815181
This is using Win 2003 Std.
0
 

Author Comment

by:IH666
ID: 11816396
So, there's no info on what exactly is causing the problem and whether this would be fixed in later releases of .NET?
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11818319
This article has a VERY good explanation of what is going on:
http://www.brianmadden.com/content/content.asp?id=69

-Eric
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11818339
Also, that article mentions that the bootswitch equivilant to the windows 2000 /3GB is, for windows 2003, /4GT.  This should give you a bit more to play with, so long as you are not running in a terminal services environment.

-Eric
0
 

Author Comment

by:IH666
ID: 11819024
I read through this, and as I understood from the comments, the only solution is to upgrade to 64-bit Win?

I presume any version (including Standard) would work, on any type of processor.

Right?
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 11819345


to my knowledge, .net is not supported in 64-bit Win
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 11819960
WOWing Your 32-Bit Applications with 64-Bit Windows Part 1, March 8, 2004
http://www.devx.com/amd/Article/20342

>>"Microsoft hasn't yet ported the .NET Framework to 64-bit Windows, and the 32-bit version will not run on that environment—so, you can't run any .NET apps on 64-bit Windows"

Bob
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 3

Accepted Solution

by:
bigjim2000 earned 500 total points
ID: 11821459
It looks to me like you need to get a 64-Bit computer, get 64-Bit windows, and wait a few more months for the release of 64-bit .NET framework before you can continue.  Until then, I'd suggest trying to get by with the /4GT bootswitch.

-Eric
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11821590
Also, have you considered using a process pool full of applications designed specifically to add more available memory?

If you need just the memory, and don't need it to be incredibly efficient, then you might stand a chance going that route.

It could be set up relatively easy with remoting (but it would suffer more of an overhead cost than using, say, a named pipe).  It would be even easier if each process simply held an array of a certain data-type.  At one point I was working on a project that needed an array of roughly 115 billion booleans (about 3GB).  I ran this in 3 processes, and had each one hold about 38Billion.  Of course, the apps did extra proccessing internally, so that I didn't have to index each one individually....it was a mess.... and still slow.... but it might be a route for you.

-Eric
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11821683
out of curiosity ... with this size array why not just store it in file and map views of the file directly into your memory space ?
0
 
LVL 96

Expert Comment

by:Bob Learned
ID: 11821687
Some heavy reading from Raymond Chen:  

Why is the virtual address space 4GB anyway?
http://weblogs.asp.net/oldnewthing/archive/2004/08/17/215682.aspx

Myth: The /3GB switch lets me map one giant 3GB block of memory
http://weblogs.asp.net/oldnewthing/archive/2004/08/16/215089.aspx

Myth: The /3GB switch expands the user-mode address space of all programs
http://weblogs.asp.net/oldnewthing/archive/2004/08/12/213468.aspx

Myth: You need /3GB if you have more than 2GB of physical memory
http://weblogs.asp.net/oldnewthing/archive/2004/08/11/212720.aspx

Myth: Without /3GB a single program can't allocate more than 2GB of virtual memory
http://weblogs.asp.net/oldnewthing/archive/2004/08/10/211890.aspx

Myth: Without /3GB the total amount of memory that can be allocated across all programs is 2GB
http://weblogs.asp.net/oldnewthing/archive/2004/08/09/211356.aspx

Kernel address space consequences of the /3GB switch
http://weblogs.asp.net/oldnewthing/archive/2004/08/06/209840.aspx

The oft-misunderstood /3GB switch
http://weblogs.asp.net/oldnewthing/archive/2004/08/05/208908.aspx

Bob
0
 
LVL 37

Expert Comment

by:gregoryyoung
ID: 11821737
then have a process pool that operates upon the disk backed shared memory :)
0
 
LVL 3

Expert Comment

by:bigjim2000
ID: 11821864
gregoryyoung has a good point, heh.

-Eric
0
 

Author Comment

by:IH666
ID: 11856584
/4GB still only lets the app use 1.2GB before crashing. Is that the expected behavior?
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11873476
You can obtain almost all physical memory available in the computer to your app.

Microsoft has a technology that is there for quite a long while that enables using more than 2GB of memory in a single process.
The technology is called Address Windowing Extensions (AWE):
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenlib/html/awewindata.asp

This will not enable .NET allocations of objects, but it enables allocations of unmanaged memory limited only by physical pages.
However, you cannot use all of this memory at the same time.
The basic steps of using AWE memory are:
1. Allocate physical pages associated with your process (using AllocateUserPhysicalPages)
2. Reserve an address space region (not memory) for the physical memory (using VirtualAlloc with MEM_RESERVE | MEM_PHYSICAL)
2. Map the physical pages into that address space region (using MapUserPhysicalPages)
3. <Use that memory>
4. Unmap the physical pages from the address space region (using MapUserPhysicalPages with last parameter of NULL)
5. Release the reserved address space region (using VirtualFree)
6. Free physical pages (using FreeUserPhysicalPages)

This way you can use almost all physical memory in the computer, mapping portions of it to your address space at a time.

Important Note:
To use AWE, you need the "Lock Pages In Memory" privilege. This requires editing of the computer's local security policy (since this is not enabled by default) and some extra code to enable the privilege in your process.
All of this is in the MSDN examples.

*** This technology does not require special operating systems and is supported from Windows 2000 onwards ***

Hope this helps,

Eran
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11874925
I once wrote wrappers for FileMapping which is very close to AWE but doesn't need a special privileg. You can find it here: http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=36988E96-BA51-4FD5-AF73-AD917B50DF1F
You also may use it as base to write a wrapper for AWE.
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11878124
Keep in mind that using FileMapping has two major drawbacks when compared to AWE:

In order to use FileMapping as a paging mechanism, you must have a filesystem object to back the mapped memory (either your own file or the paging file). This means that you either must create a large file, or risk enlarging the system's page file. These operations include some heavy I/O operations and may introduce a considerable amount of time.

Another important drawback is the fact that the backing store must be read into memory before you can write to that memory (except first usage of pagefile backed memory). So if you use your own file, before your code can touch a single byte of mapped memory (actually at the moment it does), the relevant page is read from the file into memory. Since you obviously read a lot, this will create a lot of I/O, not to mention that every writing of bytes into the mapped memory will also write them to the filesystem.

With AWE no implicit I/O operations are performed. You have full control over the memory.
For example, you could hold the persistant data in compressed format and decompress it into AWE memory. with FileMapping, this cannot be achieved...
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11885572
>In order to use FileMapping as a paging mechanism, you must have a filesystem object to back the mapped memory (either your own file or the paging file).
I think this is always the case (with AWE it's always the paging file) - but the bytes are not necessarily written to the file. The paging file should already be large enough so there is no extra overhead since it exists anyway. With both you can only access ~1GB of the memory at once. In case that you have 4GB memory and access 3GB of it there might be a difference in the behavior but to be sure I would have to test it. AWE will probably be better choice if it's important to avoid paging as much as possible. But be aware that you force all other applications paging their memory to disk by allocating a huge part of the physical memory. AWE is very good to access memory from a 32-Bit Process on a machine with very much memory (e.g. a 64-Bit machine with 32GB memory). AWE will help your program to run better on future machines, but it doesn't completely solve the current problem.
Be aware how memory management works under the target operating system. Make tests, then decide what you will use.
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11886427
ptmcomp:
A minor correction on your last comment.
When using AWE, the memory you allocate is a set of LOCKED physical pages (in physical memory).
It is never swapped out and therefore is NOT backed by the pagefile.

If you free the physical pages, the data is simply lost. As long as it is allocated, it is locked in memory and is not swapped out.

With memory mapped files, the physical memory used for the mapped file can be paged out and therefore needs a backing store
(there is no guarantee that the memory is physical)

So basically, AWE does not require enlarging the pagefile or any other filesystem related operations.
It is your responsibility to control the locking, contents, and release of the actual physical memory.

I know all of this because I have used AWE as a mechanism to access large volumes of data in proven projects.
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11886675
Yes, you're right about that. You can lock the physical pages (or they are even always locked). If you have only 4GB memory this means that all other processes share the left over physical memory, so if you allocate more than 80% of the physical memory the system's performance will probably be near zero. AWE is great if you have lots of physical memory. My wrappers for file mapping contains wrappers to access unmanaged memory which you could use with AWE, if you like to. But I didn't find a way to avoid the memory to be copied while accessing it.
0
 
LVL 4

Expert Comment

by:Eran_R
ID: 11886921
We prevented the copying using unsafe code and IntPtr.
When we lock a large object, we return an IntPtr that points to its memory.

We were able to simultanously access a little over 2.5GB of memory (32bit machine, 4GB RAM, /3GB flag)
0
 
LVL 10

Expert Comment

by:ptmcomp
ID: 11887067
If you need maximum memory and performance you will need unsafe code, you're right.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

We all know that functional code is the leg that any good program stands on when it comes right down to it, however, if your program lacks a good user interface your product may not have the appeal needed to keep your customers happy. This issue can…
Performance in games development is paramount: every microsecond counts to be able to do everything in less than 33ms (aiming at 16ms). C# foreach statement is one of the worst performance killers, and here I explain why.
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

26 Experts available now in Live!

Get 1:1 Help Now