Exchange Server Crash

Hi everyone, can you tell me if this makes sense and what you might try to do about it?

My environment is Server 2012 R2 on a Dell Poweredge Rack server. There is just one MS Hyper-V VM hosted on it which is running Exchange 2013 also on the Server 2012 R2 O/S. The host server is also a file share and AD, DHCP, DNS, etc. (I know that is not ideal)  There is 8 gig of ram and 1 processor assigned to the VM. The host has 32 gig total with plenty of free space. The servers have been running great for about 8 months.

The Exchange 2013 server stopped working over the weekend. By stopped working I mean mail stopped coming in and Outlook running on the desktops showed outlook as disconnected. According to the VM console it was still running and I could log into it but found the performance to be brutally slow.  So slow in fact that it took 10 minutes or more just to open mmc and look at services.  CPU was spiked at 100% There are no disk errors or anything like that in the logs of either the VM or the host.

So when the problem was first discovered. I rebooted just Exchange. Problem continued - Exchange services were not started (they did not respond in a timely fashion) and the CPU was inexplicably at 100% utilization.  The Noderunner, MMC and Task Manager were blamed.

Rebooted both the VM and the host.  No change. What could be going on with this server??

What I did find was that the server had run LOW on disk space.  The total volume size was about 250gb and there was only 17gb left. It would make sense that Exchange would shut down for this reason as it requires a certain % of free space to stay running.

I tried booting safe mode. I thought maybe that would rebuild paging files if they were corrupt, etc. No change.

Next I increased the size of the VM disk in Hyper-v, then increased the size of the C: partition (the VM has only one partition) and that left about 150gb free of 400gb. Rebooted the server and was surprised to find it in the same state!  Still no exchange, and still with a maxed out CPU.

I'm in the process of restoring a pre-crash backup of the virtual hard drive.  This will set them back a few days in email.

I don't think of this as an exchange problem because none of the exchange services are running and CPU is still maxed.  

Does anyone have any suggestions as to what I might try to restore server functionality before I take the drastic step of going to my VM Hard drive backup?  

I'm hesitant to add processor as I'm afraid that will just get sucked up to 100% too.

Thank you,

Paul
paul_lcsAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Casey WeaverManaged Services Windows Engineer IIICommented:
1 processor is a horrible idea in a VM sense. Hyper-V assigns by core, you should have a percentage view in there. I'm guessing the VM is only getting 8% of the hosts resources. On top of that, it's only 1 core available to the VM. Windows update by itself can eat one core as it's updating in the background, let alone exchange tasks like database checks and what not. My guess is you have a maintenance or update task running in the background that is completely chewing up that core which is preventing the Exchange services from getting proper CPU utilization. Bump that to at least two cores.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
paul_lcsAuthor Commented:
But Casey, I'M SCARED!  lol. What I was afraid would happen now is that if something is really running amok, I will freeze myself out of the VM host.  I guess I could set the VM to not start automatically as a safe guard.  Thanks, I'll try it and let you know how I make out.
Casey WeaverManaged Services Windows Engineer IIICommented:
How many cores do you have? As I said in the CPU area you'll have a percentage continue to climb upward as you assign more cores, just don't assign 100%. That being said, I've never had the host freeze up even on a locked VM. The host hypervisor protects itself fairly well.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

paul_lcsAuthor Commented:
Processor is a E5-2430, 6-core. So since it's the only VM would you recommend a throw like 4 core at it and spin it up and see what happens?
Casey WeaverManaged Services Windows Engineer IIICommented:
Yes that's what I would do.
paul_lcsAuthor Commented:
As soon as that backup restores, I'm going to give it a shot! Should be done about 9pm and I will post an update as soon as I try.  Thanks again for your quick response.
Casey WeaverManaged Services Windows Engineer IIICommented:
I would try with your current server, not the backup. No point in losing data if you don't need to.
paul_lcsAuthor Commented:
I gave it 6 of the 12 available processes and it is more responsive but CPU is still pegged. Many of the exchange processes are not running. Exchange RPC Client access is in a starting state. Using the CPU, according to task manager is noderunner, task manager, noderunner (again) iis worker process, service host: local system Microsoft.exchange.servicehost.
Casey WeaverManaged Services Windows Engineer IIICommented:
Are any pending windows updates installed? Probably going to have to start posting event viewer logs. Someone more versed in Exchange can probably help tell you what's going on.
paul_lcsAuthor Commented:
Hi Casey,  So it turned out that all my performance issues on the computer were related to older firmware on the BIOS and RAID controller in my Dell Power Edge R420.  It was only a few versions back but apparently some of those PE RAID controllers have a glitch where they develop slowness over time. I was skeptical but I tried it and my CPU utilization went from 99-100% to 2-5% and all the exchange services started right up.  I'm still going to accept your answer as a solution though because it really did help me get ahold of the computer when I took your advice and added more virtual processors.

Thanks again!

Best regards,

Paul
paul_lcsAuthor Commented:
Assigning more cores to the machine got it to run well enough to be able to at least work with it and troubleshoot further without waiting minutes between clicks and programs opening.
Casey WeaverManaged Services Windows Engineer IIICommented:
Glad to hear you got it figured out. A year ago I was hit by a bug like that on an HP Gen 8 server running a P410 raid controller. Here is some more info about virtual processor scheduling and assignment. I run several hosts at about 8-to-1 ratios with no issues. This is because rarely are all the VMs busy at the same time. During maintenance with windows updates, it gets busy (and you can hear the fans rev up quite a bit showing that the 2 E5 Xeons in each chassis are actually loading up their cores), but the intelligent scheduling in Server 2012 R2 takes out pretty much all the worry for most environments doing non-compute tasks. In a virtual desktop environment, 10-to-1 and 12-to-1 ratios are not at all unheard of.

http://www.virtualizationadmin.com/articles-tutorials/microsoft-hyper-v-articles/general/maximizing-your-virtual-machine-density-hyper-v-part4.html
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2012

From novice to tech pro — start learning today.