asked on

Server 2008 R2 Memory Leak

I have a new Dell R710 Server, dual quad-core processors with 16Gb physical RAM. It has the inbuilt Broadcom BCM5709C four-port network card running in teamed mode, an Intel dual-port MT network card for running four virtual machines through and another dual-port card connected to an iSCSI subnet to a SAN, running jumbo frames. (I mention this as I found some problems that hinted at jumbo frames but it didn't really match our configuration)

If I start the server up it is consuming 1.86Gb of physical memory with all the VMs shut down. The non-paged memory usage is about 140Mb. If I start VMs I see the expected memory consumption.

However, there is a leak somewhere. After about 24hours the server is completely out of memory and unresponsive. All I can do is reboot. If I leave the VMs off it still leaks but lasts for longer as there is more physical memory to consume. All I can find in the event logs at the time of crashing is an error saying the non-paged pool is exhausted.

If anybody could give me any help I'd really appreciate it as it's becoming a nightmare.....

Premkumar Yogeswaran

Hi,
Can you check this link for memory diagnasition
http://oca.microsoft.com/en/windiag.asp

cazzer

ASKER

I can run the memory test but I don't think physical memory issues would give the problems I am experiencing. I'll try it anyway.

dkumar82

Did you check the handle count of all the processes that runnings, check if any process have more handles. Use process explorer (sysinternal tool) to find out the actual thread causing the leak

I came acrosss the Non paged issue and further troubleshooting its become more complicated to understand and finally contacted PSS( Microsoft support) and they asked for crash dump.

Just reference:
http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx
http://support.microsoft.com/kb/177415

cazzer

ASKER

Can you elaborate on "Did you check the handle count of all the processes that runnings, check if any process have more handles" ?
I'm not sure what you mean.

dkumar82

You can find the Task manager. Its help to check how many handles are associated with process
img.JPG

cazzer

ASKER

OK, I can see the number of handles - but what is good and what is bad? One scvhost process has 1200 handles but is that bad??

ASKER CERTIFIED SOLUTION

dkumar82

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

cazzer

ASKER

I think I've found the problem - the network teaming driver (not visible in taskmanager) is consuming 6Gb of non-paged memory and climbing.
I've removed the teaming software and it seems to have settled. I'll report back after further observation :o)

cazzer

ASKER

The problem is now fixed. Thanks for all the answers but I'm awarding the points to dkumar82 for taking the trouble to post the links to fathoming out Poolmon results.

ITSMA

I'm having the same issue with a Dell server and I think the isue is with the Broadcom Team software. What did you do to resolve the issue? Did you just stop using the teaming software or did you update the Broadcom package?

cazzer

ASKER

I had to remove the teaming software. Dell raised my case with Broadcom and Broadcom have made a fix which will be publically available on 15/12/09

newsignature

Do have a patch number? ticket number? or any kind of reference number I could use when calling dell or broadcom? I have several r610s with this same problem, and would prefer to not have to run them all without teams.

ITSMA

There was released on 12/15 so I'm assuming this is the code to use. I've updated and so far not had issue. These were the 64 bit vers anyways. You should see them by release date under your specific product and OS.

Firmware: R246418
Driver: R245562