Virtualization performance issues

Tell me more about your disks. How many disks? What type? Disk performance is usually the slowest part of any server. If you put four or five VMs on a set of disks that barely support a single server, perceived performance is going to really suck.

RAID 5 consisting of 4 - 1tb SATA on a PERC H700 controller, 2 volumes.
The HOST OS and VMs (vhds) are all on the same volume, different partitions

So calculating your expected IOPS for this configuration:

Note each SATA 7.2K RPM drive can yield up to a maximum of 100 IOPS per drive, but have a random read/write average of 75 IOPS and an minimum throughput of approx. 65 Mbps per drive.  4 x 75 = 300 IOPS (4 x 65 Mbps = 260 Mbps) Less 75 IOPS for RAID 5 overhead = a maximum of 225 IOPS (195 Mbps) shared between your Virtual Machines.  Note these numbers will vary up or down depending on 3Gb/s or 6Gb/s SATA (Type II or III) drives RAID controller bus speed.  RAID 10 will acheive a little better write performance.  The more drives/spindles in your array the better the performance as well as faster 15K drives will reduce capacity, will increase your performance greatly with an average of 180 IOPS per drive.  Other factors include buffer size and read/write latency.  These numbers vary upon drive manufacture as well as consumer vs enterprise grade drives.

When doing virtualization drive performance is one of your biggest concerns.  Enterprise (MLC or SLC) SSDs drives on fast enterprise class controllers can provide very significant performance increase in the 10s to 100s of thousands of IOPS per drive and RAID array; in some extreme cases over a million IOPS per PCIe SLC NAND Flash drive.

My ESX servers are a minimum of dual ix core CPUs 96 to 128GB of DRAM using 8Gb Fiber Channel connected to an Enterprise SAN.  The SAN has a minimum of 9 15K drives in a RAID 5 set or 8 in RAID 10; the overall SAN is comprised of 144 Tier 1 and Tier 3 drives (15K FC vs 7.2K SATA RPM drives).  My newer shared SAN is 6G/s SAS and 8G/s FC with Tier 1: SLC SSDs 50K IOPS/drive, Tier 2: 15K, RPM Tier 3: 7.2K RPM with over 96 drives virtualized accross the shared SAN.  Note this isn't much in comparison to most Enterprise class environments.

The bottom line is there are a number of variables that play into virtualuzation performance.  So you need to research and get a good understanding of all of the key areas (Memory, CPU, Disk IO, Network, controllers, etc...) and how to appropriately right-size and architect your virtualization environment.  

I hope this helps.

Correction: PCIe NAND Flash drive RAID array with over a Million IOPS vs a single NAND Flash drive.

It helps but what I really need is to know what should be built for a small company of about 35 users that needs file sharing (100gb), Exchange server and a Xactimate (SQL based program) server with remote access. Seems like my configuration should be fine. Not that demanding on IOPS I would expect.

Some of my questions are:
Should I use a mirror SSD for the host?
Do I need to use physical disks in RAID 1 for each VM instead of VHDs on the host Volume?
If each VM has a dedicated NIC, are there some configuration settings that should be changed such as disabling the offloading and chimney?
Should I even consider virtualization in the first place?
Am I in over my head?

I will upgrade the hard drive array to the best the box will hold. Maybe if the budget allows I will add a second CPU and RAM.

It is highly unlikely that you need more CPU. I run servers with up to 144 GB of RAM and older Xeon processors and my total CPU is less than 15% with 20+ VMs supporting over 600 users. You can certainly add RAM if you feel that your VMs would benefit from more RAM or if you plan on adding more VMs.

Thank you.
As I first stated it seemed like CPU stays at 0% most of the time so how can it be slow?

So I'm still left to wonder why, if the servers aren't doing much, can the disk IO be that bad that slows everything down. Seems like it should be better than that to me.

Does Server 2012 hyper-v perform better?

A system can have the disk at 100% and the CPU be running at 1%, and the system will seem like a dog for anything that requires disk IO, such and clicking on the Start button. Your VMs will also generate a lot more write IO than you expect - I think that it's part of Windows keeping everything up to date in terms of system logs, event logs, NTFS updates, etc. You can prevent read IO by adding RAM and then caching the data, but every write IO goes to disk. Remember, your server with 4 disks in RAID 5 has maybe about 2.5 times the IOPS of your desktop if you are using SATA, but you're running a lot more on the server.

if the bottleneck is the iops then changing the cpu, adding memory, changing the o/s will result in minimal gains. You put a volkswagon bug engine in a ferrarri it still will go only as fast as if it was in a camaro.

I've been playing with this today. I have 3 VMs. One is actually in prodution and is the one I am concerned with. It runs great if it is the only one started. The other two have Server 2008 R2 fresh installs and nothing else. If I start them up, the clients and I notice a big slow down. I just expected more out of it I guess. I appreciate all you help on this.

What do you guys use to actually watch the Disk IO as its happening to see when the bottlenecks occur and what causes them?

ManageEngine.com OpManager, which has a good variety of free tools for small network environments including OpManager.  On your Windows server, Windows Performance Monitor (Aka PerfMon).  Pull up Task Manager --> then Performance tab --> at the bottom of Task Manager window click on the Advance option --> this open up an area to see more in-depth details on Memory, CPU, Network, and Disk IO usage.

Also, you can download and install ioMeter, which you can benchmark your storage performance.