Excessive paging with Exchange 2010

We have a newly installed Exchange 2010 environment that is experiencing excessive paging. The background:

Transitiong from an Exchange 2003, single server environment to Exchange 2010, with one server with the mailbox role, and another server with the CAS and Hub Transport role. All accounts have been moved to the new 2010 server, though the Exchange 2003 server is still running. We have approximately 60 mailboxes and the physical size of the database is 65GB.

Both 2010 servers are virtual machines. They both have two cores and 5GB of RAM allocated. They both consume all of the memory allocated to them, but this is as expected. We have created separate drives for them to page to, and turned paging off for the c: drive. We have seen the excessive paging regardless of whether they were writing to c: or the new drive. The page file size for both has been set to 5121MB, or size of RAM + 1MB. Our primary DC and global catalog server is a Windows 2003 box with 2 cores and 2GB of RAM. All users have Outlook 2007 or Outlook 2010.

Our problem is with the paging. It is killing system performance on both Exchange 2010 servers. Both are very sluggish and we have had some users that have gotten messages in Outlook as it hangs while waiting for it's connection to the server. We've adjusted RAM and page file size and searched for a solution, but have thus far found nothing that resolves our excessive paging problem.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Alan CoxSr. Systems Engineer | Lead Microsoft CIE Qualified FacilitatorCommented:
Exchange ram should be RAM + 10mb. Pushing minimum ram requirements. I usually start at 8GB http://technet.microsoft.com/en-us/library/dd346700.aspx

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
PMFMAuthor Commented:
I have seen that document, and was somewhat confused by it. In the chart for memory configurations, the last column is titled, 'Recommended Maximum' and says 2GB per core. But then also says (8GB minimum). The Recommended Maximum vs minimum didn't make much sense to me. I've read on other forums where users have increased their RAM to 8GB or higher and still had the same problem, but I am willing to try it.

In the past we've set paging files as 1MB over, generally just following the guidelines for required space for dump files and using that for paging. I have since read two recommendations for Exchange 2010 - one that says RAM + 10MB, and another that says RAM x 1.5. I will schedule a time to add the extra 9MB and see how it goes.

Thanks for the quick response.
Hypercat (Deb)Commented:
Another thing you might want to look at is what processes are actually causing the paging.  I have a virtual server running Exchange 2007 and recently had a similar problem.  I found that the System process was showing almost constant memory and page file usage. I never was able to determine what was causing this, although I suspect it was something to do with the virtual environment (I am using VMWare ESXi 4.1).  In any case, the only thing I could think to do was to reboot the server. This cleared up the problem and it hasn't recurred since.
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Alan CoxSr. Systems Engineer | Lead Microsoft CIE Qualified FacilitatorCommented:
Yeah, MS documentation sucks! I have a client on exchange 2010 (single server) and about 30 users. They had the same issue until they went to about 12gb of Ram. Now this was a multirole server and they have since added sharepoint and are now up to 23 gb of ram.
PMFMAuthor Commented:
Wow... that's a heavy resource requirement for a small amount of users. I'm having a hard time swallowing 14GB total memory usage for Exchange 2010 vs. the 2GB we had in Exchange 2003, for the same number of users.

I'm going to add 2GB to the mailbox server per that document you linked to along with the 8GB for the CAS server.
Alan CoxSr. Systems Engineer | Lead Microsoft CIE Qualified FacilitatorCommented:
Yes, exchange 2010 wants to do everything in ram to lower disk I/O. In fact, 2003 exchange didn't like very much ram.
PMFMAuthor Commented:
So far this morning, paging appears to have decreased from its previous levels, though it's not gone completely. I guess the answer is to keep throwing memory that the problem until it goes away?
Alan CoxSr. Systems Engineer | Lead Microsoft CIE Qualified FacilitatorCommented:
Page 6 of the attached document shows minimum requirements. For multi role- 8gb (I usually do 12).
PMFMAuthor Commented:
Thanks- I'm going to go up to 10GB this weekend and see what impact it has during the day on Monday. The paging is definitely better since increasing to 8, but I'd like to see it go away completely. I think I'll know if the problem is fixed or not by Monday afternoon either way.
PMFMAuthor Commented:
As a follow up, though this thread has been closed. We discovered the real issue behind our problem. We continued to have issue with CPU and memory usage from our CAS/Hub server and mailbox server. Both are virtual machines in vSphere. Turns out that during an upgrade to the vSphere environment, the resource allocation limit for RAM for several virtual machines was set to the amount that it had been allocated to it. By default the limit is set to unlimited.

Keep in mind, the "limit" is different from allocating RAM to a machine. When we added additional memory to the Exchange servers, they had more than what the limit was set to. This caused the vmWare Tools baloon memory driver to go out and grab the difference in memory from the limit and what had been allocated and hold it. Setting this limit back to unlimited allowed the virtual machine to use all of the memory we had allocated to it, and the problem immediately disappeared. I am not sure of a reason why the limit should ever be changed from unlimited.

Our CAS/Hub server is now using under 5GB of RAM and the CPU usage is minimal. The mailbox server is the same. Previously, both were using every bit of RAM that we gave them. Performance of both machines is like night and day from before.

I have seen on several forums where people have complained the limit was set automatically by doing an upgrade in vmWare. I have been unable to test and try to replicate the problem, so I am basing all of this on our experience and likely cause - not on actual verifiable proof that a vmWare upgrade caused those limits to be set. All I can say definitively is that no one here changed those limits manually. Some process, possibly upgrade in versions, did it without warning or notice.
Would like to add that PMFM's above solution worked in our situation too! The Exchange VM was not set to Unlimited in the Guest OS's Edit Settings ---> Memory ---> Resources Tab ----> Memory. We gave ours 40 Gig just for 108 users on a single Exchange box and it stopped locking up but ate up to 38 Gig of RAM. After applying Roll Up Patch 3v3 for SP1 the issue still carried on!

I found the above and set the Memory to Unlimited (like all the other VMs were by default) and now the Exchange box is only using 3 Gig!

To me it seems that VMware is awarding Memory exclusively to the processes Commit level when in Windows commit level doesn't mean what is actually used. Tick unlimited and our store.exe is using 2 gig but is commited to 13 gig...but not using 13 gig. Same applies to other processes such as the EdgeTransport, IIS worker etc etc etc.

Thanks PMFM!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.