I am running into one hell of a weird memory problem...
A DLL library containing mixed managed and unmanaged C++ code is trying to allocate several large arrays of memory (around 160 MB X 4). The system currently has more than 2 GB of free memory so obviously it's not a true out of memory problem. The code that requires the memory is unmanaged. A managed C# program uses the DLL library. I tested the program immediatly after Windows started so this could not possibly be a memory fragmentation problem.
I tried using VirtualAlloc(), HeapAlloc(), GlobalAlloc(). All failed at exactly 600 MB. I tried creating a new private heap using CreateHeap() to overcome what appears to be an artificial limit but that did not work. I even tried using the page file directly using CreateFileMapping() to no avail.
Using 64bit operating systems to overcome the 2GB per process limit is not an option.