Help Finding Memory Leak on Exchange Server NonPaged Pool Memory (NPP)

I have been trying to find a memory leak that keeps my Exchange server failing over to the other passive node in our Exchange Server 2003 Cluster.

The last failover was because our NPP was too high and our HTTP Virtual Server resource failed impacting webmail. This happened between 04/08/2010 and 04/09/2010

I attached my poolmon file hoping this would help. What I am most confused with is what I am supposed to be looking for in this file.

Memory leaks are ones where the DIFF is continually increasing?
tbexchmbxpdnd03.txt
thesurg3onAsked:
Who is Participating?
 
thesurg3onAuthor Commented:
ok. thanks.
0
 
Justin OwensITIL Problem ManagerCommented:
My organization has been fightingh this problem for over 2 years.  We have 6 Exchange clusters in an Active-Active-Passive config.  The clusters intermitentatly fail due to NPP errors.  The problem boils down to needing a hardware refresh.  In the meantime we have worked with our onsite MS rep to reduce the occurances.  To do this, we have scheduled weekly reboots of all the Exchange servers as well as deployed a script to turn off the Outlook To-Do Bar (on first use, users can turn it back on, but most don't) because of the utilization strain it places on the Exchange servers.  This is a proble unique to Exchange 2003 in a cluster.  The problem doesn't exist in 2007 or 2010 clusters.
Yes, the ones where you see continual growth are the memory leaks.
0
 
thesurg3onAuthor Commented:
I never heard the To Do bar as being resource intensive. Did you do this via Group Policy?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
Justin OwensITIL Problem ManagerCommented:
We do this via login script pushed by GPO.
'This sets the registry key to disable the to-do bar in Outlook.
'Once it does this, it write a flag file to the local user's profile and will never set the key again so long as that flag file exists.
Function DisableToDoBar

    On Error Resume Next
    
    If Not oFSO.FileExists(lup & "\ToDoBarTouch.txt") Then
        RegLocate = "HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\Options\ToDoBar\DisableToDoBar"
        oShell.RegWrite RegLocate,&H8000001f,"REG_DWORD"        
	    Set File = oFSO.OpenTextFile(lup & "\ToDoBarTouch.txt", ForWriting, True)
	    File.Close
    End If
    
End Function

Open in new window

0
 
MegaNuk3Commented:
Run the Exchange Best Practice Analyser:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=dbab201f-4bee-4943-ac22-e2ddbd258df3

To make sure you are not missing anything obvious.

Remind me, are you getting a 2019 or 2020 event?
0
 
MegaNuk3Commented:
Also confirm if the server has 4Gb of RAM and you are only seeing 3.5Gb in Windows?
0
 
thesurg3onAuthor Commented:
no 2019 or 2020. 4gb of ram and 3.5 shown.
0
 
MegaNuk3Commented:
Hmmm, is it a COMPAQ/HP server? If so, then you need to go into the BIOS and turn off memory hotswap.

Did you run ExBPA? Did it highlight anything?
0
 
MegaNuk3Commented:
Are you sure you aren't getting a 2019 event logged? Have a look at this article:
http://support.microsoft.com/kb/968684/en-us
0
 
MegaNuk3Commented:
Looks like you are running compaq hardware. Make sure you are using the latest drivers available because your network card driver is consuming 9Mb of Non paged pool memory (bxnd52x) which is quite a lot of Non paged pool memory.

Can you confirm your page file size? Is it 6.3Gb? That is too big. ExBPA should highlight it. What settings are you using in the boot.ini ? /3Gb and /USERVA=3030?
0
 
thesurg3onAuthor Commented:
yes. this is an HP DL380. NICs are Broadcoms, but relabeled as the following.

HP NC373i Multifunction Gigabit Server Adapter
HP NC360T PCIe Gigabit Server Adapter

and we are using HP Teaming.

Running the HP Version Control Agent, my nic drivers are not the latest, but here are the stats (images attached of out of date items. nothing stands out to fix a memory issue.

Page File (Custom Size): Initial: 3000MB Maximum: 3326MB

EXBPA only states the following:






VCA-Updates-1-of-3.jpg
0
 
thesurg3onAuthor Commented:
image 2 of 3
VCA-Updates-2-of-3.jpg
0
 
thesurg3onAuthor Commented:
image 3 of 3
VCA-Updates-3-of-3.jpg
0
 
MegaNuk3Commented:
Sorry, what does ExBPA say?

What I would do is, go into the BIOS and turn off hot swap memory support, this should also let windows see the full 4Gb of RAM. You may need the BIOS update. See if your problem still occurs after this.

If the issue still happens then. Do what ExBPA says (probably says your pageflle is smaller than the amount of RAM you have) and then if the issues continue update your NIC drivers.

0
 
thesurg3onAuthor Commented:
EXBPA information (FYI, just a standard one not the 2 hour baseline)

All are warnings

1. unknown schema extension. This is because we are going to exchange 2010
2. IMAP4 fast message retrieval. We aren't using IMAP4 at all.
3. Log buffers optimization. All of my storage groups accomodate more than 500 mailboxes. I suppose I should adjust this... This is for all of my 4 Storage Groups. BTW, we have 4 storage groups and 5 mailbox stores on each SG.. Pretty much maxed out.
"Storage group 'My_Storage_Group' on server MYEXCHANGESERVER accommodates more than 500 mailboxes. Log buffers should be increased to improve overall Exchange server performance."

4. Large Database warning. We are aware of this as well.
"Storage quotas are not implemented for mailbox store 'MY_Storage_Group_MailboxStore' on server MYEXCHANGESERVER. Implementing quotas can improve database management and operations."

5. Nonpaged pool memory warning. On my active node of the cluster
"Nonpaged pool memory on server ACTIVENODE.mydomain.com is over the warning threshold of 100 MB. This may cause system instability. Current value: 100 MB."

6. Page table entries is low
"Free page table entries (PTE) on server ACTIVENODEmydomain.com is at the warning threshold. This could cause system instability. Current free PTE count: 6096."
EXBPA-1.jpg
EXBPA-2.jpg
0
 
MegaNuk3Commented:
Sort out the 4Gb of RAM issue so that Windows can see all 4Gb not 3.5Gb cause what you are effectively doing with /3Gb and /USERVA=3030 is telling Exchnage it can have 3gb of Memory and only leaving 1Gb to the OS. the hot swap memory option will eat up nonpaged pool memory as it links into kernel mode drivers...

Also who in their right mind is not going to take the server down before swapping memory?
0
 
thesurg3onAuthor Commented:
I will do as you say, if I off the "Hot Swap Memory" option, should I leave my /3GB and /USERVA=3030 set in my boot.ini?
0
 
thesurg3onAuthor Commented:
after your help, what we plan on doing,

adjust our Log buffers in ADSIEdit, turn off the "Hot Swap Memory" option, as well as upgrade the drivers to the latest. I always thought it was the HP Teaming that took up the extra .5GB of memory. But the "Hot Swap Memory" options does seem very plausible and I really appreciate you pointing this out.

http://technet.microsoft.com/en-us/library/aa995632(EXCHG.80).aspx

Currently our values are undefined in the parameter: msExchESEParamLogBuffers
0
 
MegaNuk3Commented:
Just turning off the hot swap option will probably fix your non paged pool mem issue.
0
 
MegaNuk3Commented:
The others will be good to do too though.
the logbuffers should speed up your Exchange database performance.
0
 
MegaNuk3Commented:
yep, leave /3Gb and /USERVA=3030 in your boot.ini

you don't have /PAE in there do you?
0
 
thesurg3onAuthor Commented:
We do not have /PAE
0
 
MegaNuk3Commented:
good
0
 
thesurg3onAuthor Commented:
hi i have to wait until next weekend to try these changes, do you mind if i leave this open until then? i am most certainly going to reward you with the points, just wanted to know your thoughts on keeping it open until I can try it.
0
 
MegaNuk3Commented:
Yep, leave it open.

Don't award points until you get solutions ;-)
0
 
thesurg3onAuthor Commented:
leaving open until this weekend.
0
 
MegaNuk3Commented:
I am looking forward to the result. How quickly from a reboot do you normally have to wait before the Non paged pool is deleted?
0
 
MegaNuk3Commented:
any update?
0
 
thesurg3onAuthor Commented:
I couldn't find the BIOS setting for hotswap memory. I went through every settings.
0
 
Justin OwensITIL Problem ManagerCommented:
What server model do you have?
0
 
thesurg3onAuthor Commented:
HP DL380 G5
0
 
MegaNuk3Commented:
Can you see if the below key exists:
HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\DynamicMemory (REG_DWORD) 1

If it doesn't exist then consider adding it as this will turn off hot swap memory support from an OS level.

As per: http://msexchangeteam.com/archive/2005/12/14/416065.aspx
0
 
thesurg3onAuthor Commented:
no. no dynamic memory support.


memory-management.jpg
0
 
MegaNuk3Commented:
Add the key as that will disable hot memory support
0
 
MegaNuk3Commented:
Don't close the question until the problem is resolved...

Is it resolved?
0
 
MegaNuk3Commented:
Errr, thanks for the points... Did the reg key resolve the leak?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.