VMware ESXi 5 Host to Guest & Guest to Guest (VM Switch) Traffic Slow

Hi Guys -

One of our customer sites runs an ESXi 5 server with SBS2008 and Windows Server 2008 x64 on virtual - with a copy of Acronis VMprotect looking after them.

Guest to LAN client seems to be fine - speed is acceptable and using a simple tool like LAN Speed Test we get about 500Mbps throughput.

The problem is that the VMprotect (which runs as a VM on the host) suffers from poor network bandwidth - around 5Mb per second or 40-60Mbps.  The same is true when transferring or doing network transfers (SMB) between the two Guest operating systems.

The bottle neck seems to be the VM Switch (we're operating in bridged mode).

NICs are all vmx3 (except the VMprotect) and I have disabled the TaskOffload as published in many VMware KB articles.  This seems to have helped the VM to Lan Client speeds but not the cross VM switch speed?

Any ideas - looked at it all!
Jock_ScotlandAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would change the NIC in the VM (VMProtect) to VMXNET3, it's a much faster NIC, than the standard E1000 NIC.

Understand and improve the network utilization profile: Network Performance Analysis and Monitoring

One of the biggest management holes in vCenter of ESX is the vSphere Client can indicate that VM network traffic is causing a 1 GB Ethernet adapter to have a 99% utilization rate. But strangely, it doesn't display which kind of traffic is going across the virtual networks, where it came from or where it's going.

To learn which traffic is going across a virtual network, there's a free tool for vSphere: Xangati for ESX, a virtual appliance that tracks conversations on the virtual network. It's great for troubleshooting any virtual network issue, analyzing virtual desktop infrastructure and correlating vCenter performance stats with virtual network stats.

and then you can really check what traffic from your 5 VMs for free is causing network issues.

It's available as a fanastic FREE download here.

http://xangati.com/try-it-free/
0
Jock_ScotlandAuthor Commented:
Hi

VMprotect is Linux based and I don't think it is compatible with VMXNET3?  Running as "Enhanced" at the moment.

I'll take a look at the Xangati offering, its 1.2Gb but i'm not sue that its not just going to tell me what I already know?  

NICs are set to Auto negotiate and are correctly connected at 1000FdX plus the VMs are all on the same Virtual Switch which has 3 physical NICs tied to it with load balancing and redundancy enabled.  Its only VM to VM traffic which seems to be affected by this?

0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
VMXNET3 is a high performing NIC.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
check networking for the VMprotect VM, what's the utilisation?
0
Jock_ScotlandAuthor Commented:
The VMprotect VM is running an average of 9431Kbps throughput (roughly 9 megabytes per second) - hope that helps....
0
Jock_ScotlandAuthor Commented:
...across the VMswitch, should this not be about 10 times that figure?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Never seen a bottleneck in the vSwitch, but we've have seen poor performing VMs.

What is the underlying datastore which the VMs are stored on?

What is VMProtect copying to?

e.g. the store?
0
Jock_ScotlandAuthor Commented:
The host server is an IBM x3550 Series with an Internal RAID5 Array, the data store is on here so should be pretty quick.

VMprotect is copying to a Netgear ReadyNAS 2100, across a Gigabit Switch.  If we copy directly from the VM to the ReadyNAS using plain old Windows then performance is fine (much, much quicker).  However copying the same file from one VM to another on the same host is equally as slow as the VMprotect to NAS.

It seems that the problem is only when copying across the Vswitch internally.....
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
RAID 5 is not the fatest performing RAID technology, when compared to RAID 10.

VMProtect to NAS
Windows VM to NAS

Both the above "should" be the same speed, but are you using the same type of NIC in both VMs?

How is the connection to NAS being made, CIFS (WIndows share), NFS?
0
Jock_ScotlandAuthor Commented:
Thr Windows NIC is a VMXNET3 and the VMprotect is an "Enhanced" - dont think Linux supports VMXNET3?

There are 3 VMs on this host, 2 of which are Windows with VMXNET3 NICs and even copying between these is slow.  Connection to the NAS is CIFS.

Puzzled.....
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, there is a big difference in performance between the Legacy NIC and the VMXNET3 NIC. The VMXNET3 NIC is connected at 10Gbe, the Legacy is 1GB.

So both VMs are not like for like.

Test performance from Windows VM to NAS, using CrystaldiskMark Benchmark
http://crystalmark.info/software/CrystalDiskMark/index-e.html

Does VMProtect support dd? and can you use dd to benchmark on the CIFS share?

Do you have a Caching Write Controller on the RAID card in thre IBM Server?
0
Jock_ScotlandAuthor Commented:
I'll try the CrystaldiskMark setup and report back.  What is dd?

The RAID controler is an IBM servRAID with I think an 256Mb cache?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
how has the cache been configured, if it's possible to configure? Is Write configured?

dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data.

it might included in the VMProtect appliance
0
Jock_ScotlandAuthor Commented:
I can't see dd in the VMprotect appliance, write caching is enabled (and battery backed up) on the RAID card.  

Still have not checked the Disk performance but I'm convinced this is not the issue, it's VMware switch throughput which seems to be the trouble.....

Jock
0
Jock_ScotlandAuthor Commented:
Ive still got the same performance issues and so far the solution has been to convert the worst offender (a SQL server)  into a physical machine again.  This has solved the immediate performance issue.  

Looking back it seems SQL server on VMWARE is not a good combination in terms of performance (from what I have read) in any case.  Ill be weary of VMWare in future!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Some SQL servers do not perform well virtualised.
0
Jock_ScotlandAuthor Commented:
We never really found a solution to this issue other than converting the machine back to physical and as such I have accepted my own solution.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.