Resource Monitor shows 4GB total "In Use" memory, but processes only account for 1GB

We have a production web server running Windows Server 2008 x64 with 8GB of RAM.  After it was placed into production the app team noticed that "In Use" memory as identified by Windows Resource Monitor (and RAMMap) has been growing over time, despite the server not being under increased load.  "In Use" memory even increases during non-business hours when the server isn't in use at all.  Currently "In Use" memory totals approximately 4GB.  However, if I look at the Memory tab of Resource Monitor and total up the Working Set of each process they're only using about 1GB total.  We want to identify the source of the memory consumption growth so it can be fixed.  

I've done a bunch of research online and came across RAMMap.  It tells the same story, though in a bit more detail (see attached screen shot).  RAMMap shows Active memory = 4,293,268kb.  However, if I go to RAMMap's processes tab and total up the memory used by all the processes it's only about 1GB.  So we have 3GB of RAM usage that is not accounted for.  Comparing this server to the identical server in our dev/test environment (which doesn't have this problem) I note that the biggest difference in memory usage according to RAMMap is Active Page Pool memory.  On the dev/test server it's only about 150MB but on the production server it's almost 2.5GB.  

Doing some reading in Russinovich's excellent Windows Internals he states there are two sitations where memory can be used but won't be included in the working set:
* Address Windowing Extension (AWE)
* Large page memory

From RAMMap we can clearly see that AWE is using no memory whatsoever, so that isn't the smoking gun.

According to Technet (http://technet.microsoft.com/en-us/query/aa366720) Large page memory will show up as part of the process private bytes, which I can also see from RAMMap and doesn't differ significantly from the working set (at least not enough to account for 3GB of missing memory!)

So at this point I'm stumped.  Any help would be greatly appreciated! RAMMap
jamesswardAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

theSAPPHIRECommented:
In RamMap, go to the FILE DETAILS Tab and review what is "loaded" and running in memory.

Also, if you notice, your MAPPED FILE area is 3.4 gigs or so, that is set aside and only 176k of it is active.. but the items are loaded and "allocated" in Memory..

let us know what you find

-sapphire
0
jamesswardAuthor Commented:
Sapphire, thanks for the reply.

Mapped File is "Shareable memory that represents a file on disk", so by definition I would expect that category to be mostly standby memory.

On the File Details tab, the file listed as consuming the most memory is using 118MB.  If I total up memory used by all files it is around 250MB.
0
jamesswardAuthor Commented:
Yesterday we took some scheduled down time to restart each service on the server one by one on the theory that one of those services would release about 2.5GB of RAM.  After restarting every running service that can be manually restarted "In Use" memory was still at about 3.7GB.  We then rebooted the server to get memory consumption back down to normal (somewhere between 800MB and 1.5GB).  We were surprised to discover that the server was still using 2.8GB after a reboot.  We're now comparing the dev/test server to the prod server to look for any differences that might explain why the dev/test system uses only 800MB when not under load but the production server uses 2.8GB.
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

jamesswardAuthor Commented:
Memory usage appears to have stabilized around 4GB.  We were unable to determine why it increased so much in the first place.
0
jamesswardAuthor Commented:
I've requested that this question be deleted for the following reason:

Problem has not been solved, and I am no longer actively investigating it.
0
jamesswardAuthor Commented:
I may have found the cause of this issue.
0
jamesswardAuthor Commented:
Finally tracked down the cause of this issue.  The server in question runs an app which generates print jobs which are then sent to other print servers around the organization.  The spooler on this server was writing a huge number of values into the HKEY_Users\.DEFAULT\Printers\DevModesPerUser and HKEY_Users\.DEFAULT\Printers\DevModes2 registry keys.  This caused the file C:\Windows\System32\config\DEFAULT, which stores the HKEY_Users\.DEFAULT\ portion of the registry to grow to it's maximum size of 2GB.  The massive growth in this portion of the registry was the source of our increased memory consumption.  It also caused other problems such as not being able to load user profiles.

Fix action: Change permissions on the DevModes2 and DevModesPerUser registry keys to deny SYSTEM the Set Value and Create Subkey permissions.  This prevented the registry hive from growing.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jamesswardAuthor Commented:
No one on EE identified the solution, which we eventually found on our own.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.