Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


VMware ESXi 5 Host to Guest & Guest to Guest (VM Switch) Traffic Slow

Posted on 2011-10-12
Medium Priority
Last Modified: 2014-01-11
Hi Guys -

One of our customer sites runs an ESXi 5 server with SBS2008 and Windows Server 2008 x64 on virtual - with a copy of Acronis VMprotect looking after them.

Guest to LAN client seems to be fine - speed is acceptable and using a simple tool like LAN Speed Test we get about 500Mbps throughput.

The problem is that the VMprotect (which runs as a VM on the host) suffers from poor network bandwidth - around 5Mb per second or 40-60Mbps.  The same is true when transferring or doing network transfers (SMB) between the two Guest operating systems.

The bottle neck seems to be the VM Switch (we're operating in bridged mode).

NICs are all vmx3 (except the VMprotect) and I have disabled the TaskOffload as published in many VMware KB articles.  This seems to have helped the VM to Lan Client speeds but not the cross VM switch speed?

Any ideas - looked at it all!
Question by:Jock_Scotland
  • 9
  • 8
LVL 125
ID: 36956217
I would change the NIC in the VM (VMProtect) to VMXNET3, it's a much faster NIC, than the standard E1000 NIC.

Understand and improve the network utilization profile: Network Performance Analysis and Monitoring

One of the biggest management holes in vCenter of ESX is the vSphere Client can indicate that VM network traffic is causing a 1 GB Ethernet adapter to have a 99% utilization rate. But strangely, it doesn't display which kind of traffic is going across the virtual networks, where it came from or where it's going.

To learn which traffic is going across a virtual network, there's a free tool for vSphere: Xangati for ESX, a virtual appliance that tracks conversations on the virtual network. It's great for troubleshooting any virtual network issue, analyzing virtual desktop infrastructure and correlating vCenter performance stats with virtual network stats.

and then you can really check what traffic from your 5 VMs for free is causing network issues.

It's available as a fanastic FREE download here.


Author Comment

ID: 36956357

VMprotect is Linux based and I don't think it is compatible with VMXNET3?  Running as "Enhanced" at the moment.

I'll take a look at the Xangati offering, its 1.2Gb but i'm not sue that its not just going to tell me what I already know?  

NICs are set to Auto negotiate and are correctly connected at 1000FdX plus the VMs are all on the same Virtual Switch which has 3 physical NICs tied to it with load balancing and redundancy enabled.  Its only VM to VM traffic which seems to be affected by this?

LVL 125
ID: 36956444
VMXNET3 is a high performing NIC.
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

LVL 125
ID: 36956450
check networking for the VMprotect VM, what's the utilisation?

Author Comment

ID: 36956515
The VMprotect VM is running an average of 9431Kbps throughput (roughly 9 megabytes per second) - hope that helps....

Author Comment

ID: 36995681
...across the VMswitch, should this not be about 10 times that figure?
LVL 125
ID: 36995838
Never seen a bottleneck in the vSwitch, but we've have seen poor performing VMs.

What is the underlying datastore which the VMs are stored on?

What is VMProtect copying to?

e.g. the store?

Author Comment

ID: 36999596
The host server is an IBM x3550 Series with an Internal RAID5 Array, the data store is on here so should be pretty quick.

VMprotect is copying to a Netgear ReadyNAS 2100, across a Gigabit Switch.  If we copy directly from the VM to the ReadyNAS using plain old Windows then performance is fine (much, much quicker).  However copying the same file from one VM to another on the same host is equally as slow as the VMprotect to NAS.

It seems that the problem is only when copying across the Vswitch internally.....
LVL 125
ID: 36999687
RAID 5 is not the fatest performing RAID technology, when compared to RAID 10.

VMProtect to NAS
Windows VM to NAS

Both the above "should" be the same speed, but are you using the same type of NIC in both VMs?

How is the connection to NAS being made, CIFS (WIndows share), NFS?

Author Comment

ID: 37000865
Thr Windows NIC is a VMXNET3 and the VMprotect is an "Enhanced" - dont think Linux supports VMXNET3?

There are 3 VMs on this host, 2 of which are Windows with VMXNET3 NICs and even copying between these is slow.  Connection to the NAS is CIFS.

LVL 125
ID: 37000949
Okay, there is a big difference in performance between the Legacy NIC and the VMXNET3 NIC. The VMXNET3 NIC is connected at 10Gbe, the Legacy is 1GB.

So both VMs are not like for like.

Test performance from Windows VM to NAS, using CrystaldiskMark Benchmark

Does VMProtect support dd? and can you use dd to benchmark on the CIFS share?

Do you have a Caching Write Controller on the RAID card in thre IBM Server?

Author Comment

ID: 37000985
I'll try the CrystaldiskMark setup and report back.  What is dd?

The RAID controler is an IBM servRAID with I think an 256Mb cache?
LVL 125
ID: 37001097
how has the cache been configured, if it's possible to configure? Is Write configured?

dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data.

it might included in the VMProtect appliance

Author Comment

ID: 37464266
I can't see dd in the VMprotect appliance, write caching is enabled (and battery backed up) on the RAID card.  

Still have not checked the Disk performance but I'm convinced this is not the issue, it's VMware switch throughput which seems to be the trouble.....


Accepted Solution

Jock_Scotland earned 0 total points
ID: 39732187
Ive still got the same performance issues and so far the solution has been to convert the worst offender (a SQL server)  into a physical machine again.  This has solved the immediate performance issue.  

Looking back it seems SQL server on VMWARE is not a good combination in terms of performance (from what I have read) in any case.  Ill be weary of VMWare in future!
LVL 125
ID: 39732251
Some SQL servers do not perform well virtualised.

Author Closing Comment

ID: 39773098
We never really found a solution to this issue other than converting the machine back to physical and as such I have accepted my own solution.

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
This article outlines why you need to choose a backup solution that protects your entire environment – including your VMware ESXi and Microsoft Hyper-V virtualization hosts – not just your virtual machines.
This tutorial will walk an individual through the steps necessary to join and promote the first Windows Server 2012 domain controller into an Active Directory environment running on Windows Server 2008. Determine the location of the FSMO roles by lo…
There are cases when e.g. an IT administrator wants to have full access and view into selected mailboxes on Exchange server, directly from his own email account in Outlook or Outlook Web Access. This proves useful when for example administrator want…

580 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question