how to track down high CPU on a 2008 server

hello experts we have a 2008 server that is hosting 3 virtual machines suddenly start having increased CPU at 25-50%. This servers virtual machines are very slow to respond. Logging in was slow and each click took several seconds. All the hard drives on the dell server are green. Task manager shows a few items with about 10%, nothing really high. Whats the best way to get to the bottom of an issue like this?
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Start with Process Explorer and Process Monitor.  Both are free tools from MS.

"Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity. It combines the features of two legacy Sysinternals utilities, Filemon and Regmon, and adds an extensive list of enhancements including rich and non-destructive filtering, comprehensive event properties such session IDs and user names, reliable process information, full thread stacks with integrated symbol support for each operation, simultaneous logging to a file, and much more. Its uniquely powerful features will make Process Monitor a core utility in your system troubleshooting and malware hunting toolkit."

Use the process monitor to see if there is anything cpu intensive running then use the process explorer as an additional tool if you need to look at a specific process in detail.  Please refer to the process monitor manual for more info.

If all the VM are slowing down due to the host, do you have any snapshots running on the host?  Also, for a start turn off all unnecessary software for troubleshooting purposes on host and clients (antivirus, malware protection, etc.)

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
What is your setup? What do the VMs do? Check whether the hosts is struggling with IO or is paging .

Specs of host memory, CPU drives.
WAMSINCAuthor Commented:
I checked for snapshots and nothing running. I had an old snapshot on one machine that I deleted and no effect. I turned off all VM's and rebooted a couple times and still very slow on each click. Im trying to get process monitor on there now.
Protecting & Securing Your Critical Data

Considering 93 percent of companies file for bankruptcy within 12 months of a disaster that blocked access to their data for 10 days or more, planning for the worst is just smart business. Learn how Acronis Backup integrates security at every stage

Christopher ScheneSystem Engineer/Software EngineerCommented:
It is very important that any VM's that you are using to run servers have sufficient memory. I had a need to run a pair of WebSphere servers, An AD Node, an SQL server and a desk top all on VM's in the same host PC. I allocated 4 Gb to each WebsSphere VM, 4Gb to the SQL 2008  server and 1 GB to the AD server---a total of 14 GB just for the VM's. I had to bump the memory on my host laptop to 24 GB: before I bumped the memory the VM's ran so slowly that it appeared that they had crashed.

For most modern CPU's you should have plenty of CPU power (I have 8 cores) and with VM's it is almost always  a memory bottleneck.  The real problem with insufficient memory is that the VM's end up thrashing in swap space.

Do make sure the host CPU is not bogged down but that is rarely the case unless there is an infinite loop someplace or you are running something really pegs the CPU like compiling a large Visual Studio project.
WAMSINCAuthor Commented:
this is a dell r520 with 65GB of RAM and 12 CPUs Running 2008 R2.
WAMSINCAuthor Commented:
Task man shows CPU at 25-40% after shutting down all VM's
This question is vague because it isn't asking about a specific service or program using the CPU. Also at 25-40% CPU I'd lean towards I/O being the issue. I can spit ball some ideas for you:

Often times with the systems like this that host VMs I see:

1. Updates haven't been done and sometimes this is an issue
2. A hard disk or RAID card is starting to head south
3. Someone decided to run any services or programs at all on the host
4. Backups cause a lot of problems when using VM aware backups in Hyper-V (some versions, but especially 2008/2008R2 with acronis) that rely on cascading VSS.
5. Someone took snapshots or made a differencing disk and is overloading the I/O

What are the full specs?
Disk Speed and type (RPM/SSD)?
RAID level (disk/vol/level/card model, #luns)?
Boot Drive?
Processor Speed and Model/Generation?
Full windows version and build number (cmd -> ver)

You might setup some performance monitors to see more detail.
WAMSINCAuthor Commented:
so mysteriously the cpu spikes are gone and its staying around 9 % now. It was acting like a failed disk but all the lights were green. Now I can move around the OS faster and clicks dont take forever. Im concerned this could happen again
Look at scheduled tasks.  Depending on what your VMs do, data intensive application, exchange, SQL, data aggregates and begins to consume more and more resources before the first access is even seen.
Underestimating the resource requirements might explain I.e. Obe VM experienced higher than expected demand.
Scott CSenior EngineerCommented:
If/when it happens again get a perfmon.

Then run it through PAL and see what if anything turns up.

Look at the report and see what is going on.

Run it on your VMs as well to see if they are doing anything.
WAMSINCAuthor Commented:
I had all the VMs off and it was still spiking CPU. I will check out PAL. Im looking at AV and windows updates. Task Scheduler was a good idea Im looking at the history on that as well.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.