Memory error in a Windows service application (Win32 project) running on a Windows 2003 x64 Ed OS

Problem : fault error on a Win32 service application running on a W 2003 X64 OS

Environment:
Server: DELL PowerEdge 2850,Processor Intel EM64T 3GHz, with 13,3 RAM physical memory available
OS: Windows 2003 Standard x64 Edition
The server is dedicated to our software
Our software:
It is a search engine for Intranets, made of a classical dll (NoeBase.dll) running as a Windows service under the NoeService.exe application.
Both programs have been developed on Visual Studio 6 as Win32 projects which have worked well on 32bit W2003 with 2 GB or RAM, till they ran on a memory problem when indexing more than a certain amount of documents (say 200,000).
We decided to port it to a WIN64 project under Visual Studio 2005, to run them on the above DELL server.
Since we had a tight schedule, we chose to take a first step by recompiling both programs under Visual Studio 2005 as Win32 projects, linking them with /LARGEADDRESSAWARE setting in order to be able to use up to 4 GB of RAM.
NoeBase is located on C:\Windows\SysWOW64
NoeBase starts by allocating a heap of 128 MB, and reallocates the heap with chunks of more memory when needed.

The problem:
The NoeService application works well but runs into an error, as follows (DrWatson message)
"The application, NoeService.exe, generated an application error
The exception generated was c0000005 at address 7D611952 (ntdll!ExpInterlockedPopEntrySListFault)"
This error terminates the service.

From our app log file, the problem occurred in our function which reallocates the heap by calling HeapReAlloc
Pointer ResizeMem (Pointer AddrLpMem, indeX size)
{
      return deRef(AddrLpMem) = HeapReAlloc(_g_MemHeap, HEAP_GENERATE_EXCEPTIONS,
                              deRef(AddrLpMem),(DWORD)size);

}//ResizeMem

_g_MemHeap is a global handle obtained by call to GetProcessHeap()
Pointer is a void * type, and indeX an unsigned long type.
AddLpMem is the pointer to the block to reallocate.
deRef is a macro: #define deRef(P)      (*(Pointer *)(P))

The problem occurs when the Commit Charge is about 1,4 GB (of which approximately 900 MB used by NoeService).

Questions : How to prevent the error, either at the code level, or at the system level ?

This is a very important problem for us, and I am willing to give a high level of points.
Daniel VerneyConsultantAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ZoppoCommented:
Hi verdan,

it could be that simply the process runs out of memory: A 32-bit process can't ever use more than 3GB of memory address range, no matter if it runs on 64-bit Windows or on 32-bit Windows-Server with '/3GB' switch.

Now imagine your service uses already about 1.4 GB - then the 'HeapReAlloc' may fail since the realloc in case of increasing size maybe can't be done by just reserving new memory to the end of the existing memory block - in this case a new memory block of the needed size is allocated, the data from the old memory block is copied to the new one, then the old one is released.

So, with memory-size of about 1.4 GB in this case the application needs nearly 3 GB to realloc the memory.

If you have to handle such amout of data I guess there are only two possibilities:

1. Compile it as a 64-bit application. This should solve the problem, at least as long as you don't allocate much more memory than the physical RAM of the machine.
2. Instead of reallocating the memory you could write the data to the disk, release the memory, allocate new memory and load the data from the disk - this will of course be a matter of performance.

Hope that helps,

ZOPPO
0
Daniel VerneyConsultantAuthor Commented:

Hi Zoppo,
If I understand your explanation, the realloc process needs more than twice the initial size, and I'had guessed that was something like that
I'm aware that solution 1 is the best, it's just a question of the time needed to do it
Solution 2 is actually possible with a little programming
However, I thought that with the /LARGEADDRESSAWARE setting in linking, it was possible to use 4 GB of memory for a Win32 app running on a Windows 64bit OS which has RAM enough (16 Gb) - at least it's what Microsoft says in its doc: do you have some hints about that question < this could give us some more time to implement solution 1>
(sorry my french keyboard does not translate well in this text box>>>) <<
Thanks
Verdan
0
ZoppoCommented:
Hi,

well, realloc either needs the size of the original memorey block (if new size is smaller), the size of the new memory block (if new size is larger but after the original block is enough free memory to extend it to the new size) or the sum of both sides if it needs to create o copy of the original memory block.

> it was possible to use 4 GB of memory for a Win32
That's not right, a 32-bit application cannot use more than 3GB, the rest is reserved for other data like program code, loaded DLLs and I guess some more.

ZOPPO
0
Introducing Cloud Class® training courses

Tech changes fast. You can learn faster. That’s why we’re bringing professional training courses to Experts Exchange. With a subscription, you can access all the Cloud Class® courses to expand your education, prep for certifications, and get top-notch instructions.

Daniel VerneyConsultantAuthor Commented:

Hi,
Thanks Zoppo,
But what about the /LARGEADDRESSAWARE setting ?
Microsoft explicitly says that it allows Win32 apps running on 64bits OS to use up to 4GB, if physically available, see
http://support.microsoft.com/default.aspx?scid=889654
Our server has 16 GB of physical RAM
Maybe I missed something ?

VERDAN
0
ZoppoCommented:
sorry, I was wrong - on 32-bin windows a process can never address more than 3GB, but there all DLLs, code a.s.o. even resides in this 3GB. In 64-bit windows for all together 4GB are available - but this still can be a problem when reallocating huge memory blocks - in fact to allocate a 1,5 GB block there has to be at least 1,5 GB free (unfragmented) memory. This can even fail with 4GB adress space ...

ZOPPO
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Daniel VerneyConsultantAuthor Commented:
I wanted to accept all the comments that you sent me, but the Experts Exchange interface is not clear at all about accepting solutions (multiple?). What can I do ?
0
ZoppoCommented:
Hi,

you did it allright - there's no need to accept multiple comments from one expert - just if you have comments from multiple experts and find more than one helpful then you can accept one as answer and add a number of other comments as 'assisted'.

Have a nice day,

best regards,

ZOPPO
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Editors IDEs

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.