Exchange VM Freezing

Hello Experts,
We have a situation where an Exchange 2010 Server running on Windows 2008R2 within an ESXI VM is "freezing" every few days. This issue has been widely documented on the internet and seems to be a combination of above running on ESXI 4.1. We have taken many steps to rectify this issue including; building a new ISCSI SAN from local storage, removing the tape drive, introducing a new ESXI host, allocating more memory to video RAM and upgrading to ESXI 5.1 but the problem remains.

We may look to try ESXI 5.5 and a rebuild of the ESXI OS as apposed to an upgrade. But we're now at the stage where we need to consider a complete Exchange rebuild, this would involve adding a new Exchange Server, then migrating users/mailboxes across. So my question is two fold;

1. Can anyone suggest a resolution to the VM "locking up" issue where Exchange is accessible through the ESXI console but no peripheral/mouse etc movement works and everyone looses connectivity to the Server?

2. What’s the best procedure for introducing a new, replacement Exchange Server in a live environment?
markfureyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Simon Butler (Sembee)ConsultantCommented:
The only cause of freezing that I know of with VMWARE is the video driver issue. If you pick the wrong one then it causes this issue. I have been running Exchange 2010 on VMWARE since before RTM and it has been fine.

Adding a new server in to live is very easy - just ensure it is the same patch level. Remember that you can use Service Pack files to deploy a new server.

The biggest headache is moving the clients. Unless you have an RPC CAS Array in place you have to touch all of the clients to get them to use the new CAS role holder.
If you don't have an RPC CAS Array then it would be a good idea to implement one right now and start the work on getting the clients to use it (repair Account in Outlook).
Then moving the clients is easy - just update the DNS entry.

Simon.
0
pgm554Commented:
Sounds like a memory leak.

Have you tried limiting the amount of RAM Exchange can use?

http://www.bursky.net/index.php/2012/05/limit-exchange-2010-memory-use/
0
markfureyAuthor Commented:
Thanks for your comments, we have already installed the recommended, VM graphics driver and the link regarding memory issues seems to be regarding SBS servers. I think I will try a virgin install of an updated ESXI and if the problem persists then look to replace the Exchange Server. I will try to find a definitive white paper on how to implement this, I know Simon insists it's "easy" but his instructions are slightly vague. I was hoping there would be a method of migration without needing to visit every client. I know when we upgraded from Exchange 2003 this was possible; I was also hoping we might be able to move the whole Exchange datastores rather than individual mailboxes.
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

pgm554Commented:
All of the newer Exchange systems grab as much ram as they can.

If you limit the amount of memory they can allocate and the issue goes away,you know it's a memory leak.

From the symptoms you describe,it's a classic memory leak of some sorts.
0
Simon Butler (Sembee)ConsultantCommented:
There is no other method unless you visit every client that is risk free.
Microsoft's "method" is to remove the old server then cross your fingers and trust that Autodiscover works when the server has gone away.

I am not making that up.

You cannot test it because Autodiscover will only kick on when the original CAS role server has gone away. That means Exchange has been removed - shutting down is not enough.
Once it has gone, you can't put it back so the clients accept it, so if it doesn't work, you have to visit every client again.

If you had deployed an RPC CAS Array right at the start then you wouldn't have the problem.

You are quite correct that when you moved to Exchange 2010 from 2003 you didn't have to touch the clients, and Microsoft don't seem to be grasping the issue (it would seem that in their world people don't move to the same version, they just upgrade).

I have been banging on about this for years. Here is a blog posting from November 2010 where I mention the issue - that is almost three years ago!
http://blog.sembee.co.uk/post/RPC-Client-Access-Array.aspx

While you can move the databases around (Exchange supports database portability) I wouldn't recommend it, as it introduced unnecessary risk. You have to shutdown the old server, move the file, then get it to accept the newly copied database.
Or you can just have both servers running at the same time, move mailboxes with a script live, putting no data at risk and not bringing across a database that has probably been damaged by the server being reset due to the freeze.

My instructions were vague, because there really isn't very much to it.
1. Install Exchange 2010 on the new server and configure the same as live. Including SSL certificates, Receive Connectors.
2. Replicate public folders
3. Move mailboxes.
4. Change OAB generation.
5. Remove old server.

Simon.
0
markfureyAuthor Commented:
Thanks, Simon, I've been doing some research since your last comments and I'm now starting to grasp the concepts involved in replacing an existing Exchange Server, I will try to find time to "lab out" the scenario before implementing the new Server in a live system. You never know, maybe a virgin install of ESXI might fix the problem anyway; it would be nice to get to the bottom of the issue rather than replacing the whole Exchange system. I have seen many issues with the exact same problem with different fixes. My concern is that there's some weird compatibly issue that will just carry over to the new, virtualised server, that wouldn’t go down well! The odd thing is that the Server was stable for well over a year, then a mix of a recovered datastore, SAS tape/controller upgrade seemed to be the trigger for the recent "freezing" events. One thing I didnt mention earler is that when the system locks up, the only information we can glean is from ESXI perfomance tab, it shows that CPU has completely "maxed" out. I will also look to reserve memory allocation to Exchange as suggested earlier.
0
markfureyAuthor Commented:
Hello Experts,

Sorry for the late reply but I'f forgot that I still had open questions left. In the end the issue was wiht the Eset Nod AV program, we uninstalled it and the issue went away, thank god! There is now a File Server product for Server operating systems which automatically include the filters/exceptions to stop interference with various Exchange/Server porcesses.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
pgm554Commented:
So it still sounds like a memory leak,it was just the AV that was causing it.

Did the folks at Eset have any solution?

I had an AVG scanner that only had a memory leak with SBS 2003 r2 and nothing else.

So it sounds like you got something that's undocumented.
0
pgm554Commented:
Which version of  ESET ?

I talked to one of the  ESET  SE's at RSA today and was told Nod 32 is no longer supported on the server products.

Only ESET File Security is.
0
markfureyAuthor Commented:
I found the solution.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.