VMware ESXi 5 Host to Guest & Guest to Guest (VM Switch) Traffic Slow
Hi Guys -
One of our customer sites runs an ESXi 5 server with SBS2008 and Windows Server 2008 x64 on virtual - with a copy of Acronis VMprotect looking after them.
Guest to LAN client seems to be fine - speed is acceptable and using a simple tool like LAN Speed Test we get about 500Mbps throughput.
The problem is that the VMprotect (which runs as a VM on the host) suffers from poor network bandwidth - around 5Mb per second or 40-60Mbps. The same is true when transferring or doing network transfers (SMB) between the two Guest operating systems.
The bottle neck seems to be the VM Switch (we're operating in bridged mode).
NICs are all vmx3 (except the VMprotect) and I have disabled the TaskOffload as published in many VMware KB articles. This seems to have helped the VM to Lan Client speeds but not the cross VM switch speed?
Any ideas - looked at it all!
One of our customer sites runs an ESXi 5 server with SBS2008 and Windows Server 2008 x64 on virtual - with a copy of Acronis VMprotect looking after them.
Guest to LAN client seems to be fine - speed is acceptable and using a simple tool like LAN Speed Test we get about 500Mbps throughput.
The problem is that the VMprotect (which runs as a VM on the host) suffers from poor network bandwidth - around 5Mb per second or 40-60Mbps. The same is true when transferring or doing network transfers (SMB) between the two Guest operating systems.
The bottle neck seems to be the VM Switch (we're operating in bridged mode).
NICs are all vmx3 (except the VMprotect) and I have disabled the TaskOffload as published in many VMware KB articles. This seems to have helped the VM to Lan Client speeds but not the cross VM switch speed?
Any ideas - looked at it all!
ASKER
Hi
VMprotect is Linux based and I don't think it is compatible with VMXNET3? Running as "Enhanced" at the moment.
I'll take a look at the Xangati offering, its 1.2Gb but i'm not sue that its not just going to tell me what I already know?
NICs are set to Auto negotiate and are correctly connected at 1000FdX plus the VMs are all on the same Virtual Switch which has 3 physical NICs tied to it with load balancing and redundancy enabled. Its only VM to VM traffic which seems to be affected by this?
VMprotect is Linux based and I don't think it is compatible with VMXNET3? Running as "Enhanced" at the moment.
I'll take a look at the Xangati offering, its 1.2Gb but i'm not sue that its not just going to tell me what I already know?
NICs are set to Auto negotiate and are correctly connected at 1000FdX plus the VMs are all on the same Virtual Switch which has 3 physical NICs tied to it with load balancing and redundancy enabled. Its only VM to VM traffic which seems to be affected by this?
VMXNET3 is a high performing NIC.
check networking for the VMprotect VM, what's the utilisation?
ASKER
The VMprotect VM is running an average of 9431Kbps throughput (roughly 9 megabytes per second) - hope that helps....
ASKER
...across the VMswitch, should this not be about 10 times that figure?
Never seen a bottleneck in the vSwitch, but we've have seen poor performing VMs.
What is the underlying datastore which the VMs are stored on?
What is VMProtect copying to?
e.g. the store?
What is the underlying datastore which the VMs are stored on?
What is VMProtect copying to?
e.g. the store?
ASKER
The host server is an IBM x3550 Series with an Internal RAID5 Array, the data store is on here so should be pretty quick.
VMprotect is copying to a Netgear ReadyNAS 2100, across a Gigabit Switch. If we copy directly from the VM to the ReadyNAS using plain old Windows then performance is fine (much, much quicker). However copying the same file from one VM to another on the same host is equally as slow as the VMprotect to NAS.
It seems that the problem is only when copying across the Vswitch internally.....
VMprotect is copying to a Netgear ReadyNAS 2100, across a Gigabit Switch. If we copy directly from the VM to the ReadyNAS using plain old Windows then performance is fine (much, much quicker). However copying the same file from one VM to another on the same host is equally as slow as the VMprotect to NAS.
It seems that the problem is only when copying across the Vswitch internally.....
RAID 5 is not the fatest performing RAID technology, when compared to RAID 10.
VMProtect to NAS
Windows VM to NAS
Both the above "should" be the same speed, but are you using the same type of NIC in both VMs?
How is the connection to NAS being made, CIFS (WIndows share), NFS?
VMProtect to NAS
Windows VM to NAS
Both the above "should" be the same speed, but are you using the same type of NIC in both VMs?
How is the connection to NAS being made, CIFS (WIndows share), NFS?
ASKER
Thr Windows NIC is a VMXNET3 and the VMprotect is an "Enhanced" - dont think Linux supports VMXNET3?
There are 3 VMs on this host, 2 of which are Windows with VMXNET3 NICs and even copying between these is slow. Connection to the NAS is CIFS.
Puzzled.....
There are 3 VMs on this host, 2 of which are Windows with VMXNET3 NICs and even copying between these is slow. Connection to the NAS is CIFS.
Puzzled.....
Okay, there is a big difference in performance between the Legacy NIC and the VMXNET3 NIC. The VMXNET3 NIC is connected at 10Gbe, the Legacy is 1GB.
So both VMs are not like for like.
Test performance from Windows VM to NAS, using CrystaldiskMark Benchmark
http://crystalmark.info/software/CrystalDiskMark/index-e.html
Does VMProtect support dd? and can you use dd to benchmark on the CIFS share?
Do you have a Caching Write Controller on the RAID card in thre IBM Server?
So both VMs are not like for like.
Test performance from Windows VM to NAS, using CrystaldiskMark Benchmark
http://crystalmark.info/software/CrystalDiskMark/index-e.html
Does VMProtect support dd? and can you use dd to benchmark on the CIFS share?
Do you have a Caching Write Controller on the RAID card in thre IBM Server?
ASKER
I'll try the CrystaldiskMark setup and report back. What is dd?
The RAID controler is an IBM servRAID with I think an 256Mb cache?
The RAID controler is an IBM servRAID with I think an 256Mb cache?
how has the cache been configured, if it's possible to configure? Is Write configured?
dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data.
it might included in the VMProtect appliance
dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data.
it might included in the VMProtect appliance
ASKER
I can't see dd in the VMprotect appliance, write caching is enabled (and battery backed up) on the RAID card.
Still have not checked the Disk performance but I'm convinced this is not the issue, it's VMware switch throughput which seems to be the trouble.....
Jock
Still have not checked the Disk performance but I'm convinced this is not the issue, it's VMware switch throughput which seems to be the trouble.....
Jock
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Some SQL servers do not perform well virtualised.
ASKER
We never really found a solution to this issue other than converting the machine back to physical and as such I have accepted my own solution.
Understand and improve the network utilization profile: Network Performance Analysis and Monitoring
One of the biggest management holes in vCenter of ESX is the vSphere Client can indicate that VM network traffic is causing a 1 GB Ethernet adapter to have a 99% utilization rate. But strangely, it doesn't display which kind of traffic is going across the virtual networks, where it came from or where it's going.
To learn which traffic is going across a virtual network, there's a free tool for vSphere: Xangati for ESX, a virtual appliance that tracks conversations on the virtual network. It's great for troubleshooting any virtual network issue, analyzing virtual desktop infrastructure and correlating vCenter performance stats with virtual network stats.
and then you can really check what traffic from your 5 VMs for free is causing network issues.
It's available as a fanastic FREE download here.
http://xangati.com/try-it-free/