Solved

transfer rate between esxi hosts very slow

Posted on 2013-06-14
21
4,241 Views
Last Modified: 2016-11-23
Dear experts,
I have a strange problem with speed between two ESXi 5.1 hosts managed by a vCenter.
The machines involved are two Dell PowerEdge 2950 (local datastore, one has 15k disks, the other 7k2) , connected together by redundant gigabit network.

Each of them has a local datastore on which runs some VMs.

If I do a file transfer between this hosts by Datastore Browser or by a backup software (VMX emplorer), the speed that I reach is about 6/10 MB/sec.

You might think of a problem resident on the network, but performing a file transfer between VMs runnings on different host , I reach good performance (60/100 MB/sec). I also try to disable redundant nic for sake, but problem stiil remains.

We may think of datastore I/O problem of one of the two hosts, but it is not so because benchmarking each datastore, each result is good.

Any suggesitons?

thank's a lot

andrea
0
Comment
Question by:Andrea_Corbo
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 11
  • 7
  • 2
  • +1
21 Comments
 
LVL 121
ID: 39247523
it's likely to be caused by the speed of reading and writing to VMFS partitions.
0
 

Author Comment

by:Andrea_Corbo
ID: 39247570
...may be...
I try to give you even this element: I also have a nfs shared storage (very cheap, qnap basi model).
From and to this nfs storage,  the speed reached by each ESXi hosts is the maximum possible for this nfs storage (30MB/sec).

So I remain confused...
0
 
LVL 121
ID: 39247616
are jumbo frames enabled on your VMKernel?

are the RAID and disk types the same?
0
When ransomware hits your clients, what do you do?

MSPs: Endpoint security isn’t enough to prevent ransomware.
As the impact and severity of crypto ransomware attacks has grown, Webroot has fought back, not just by building a next-gen endpoint solution capable of preventing ransomware attacks but also by being a thought leader.

 

Author Comment

by:Andrea_Corbo
ID: 39247741
I just setted up MTU on VSwitch  at 9000 (also anabled in phisic HP switch).

NOW file transfer speed from server B to server A is back to over 100MB/sec, but if i do inverse operation, from SERVER B to A (copying the same file), speed is again slow, at 10MB/sec...

the raid is 6 type on both servers.

thank's a lot for your support
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 500 total points
ID: 39249376
How are you doing the file transfer?

Do a packet capture for just a few seconds in each direction.  Verify that jumbo frames are being use in both directions.  If the file transfer method uses TCP, verify the window size is the same, or close, in both directions.
0
 

Author Comment

by:Andrea_Corbo
ID: 39249384
errata corrige to the previous post: speed problem persists.

I have tried and tested for many hours (iperf, sqlio, etc.), but  any host to host transfers is slow  (vmotion, clone, vSphere replication, past / copy from the datastore browser) from 5/6MB to 15MB/sec.

On Monday I'm going to change / try new switches.

I will update you, thank's for now....
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39251553
What is the RTT if you ping the hosts from each other?
0
 

Author Comment

by:Andrea_Corbo
ID: 39251880
Ping from host 10.1.1.204 to 10.1.1.206 (dell server)

PING 10.1.1.206 (10.1.1.206): 56 data bytes
64 bytes from 10.1.1.206: icmp_seq=0 ttl=64 time=2.950 ms
64 bytes from 10.1.1.206: icmp_seq=1 ttl=64 time=0.620 ms
64 bytes from 10.1.1.206: icmp_seq=2 ttl=64 time=0.371 ms

consider that between this hosts is now running a backup job and vmware replication....
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39251936
So no obvious errors on the pings.  A packet capture, just a few seconds in each direction, may show what is going on.

What is the CPU utilization like?  If the CPU's are being maxed out, it will affect network transfer rates.
0
 

Author Comment

by:Andrea_Corbo
ID: 39252393
Cpus and RAM in both server are very low. About packet capture, is there something embedded on esxi host console or do I mirror eth port on my switch and get data?

thanks
0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
ID: 39254435
I beleive that tcpdump should exist on esxi.

You can capture the traffic and write it to a file, then transfer the file to your local computer and use Wireshark to look at the capture.

Something like:

tcpdump -s 0 -i xxxxx -w file01.cap

The character that follows the -s is the number zero.

where xxxxx is the name of the interface you want to capture on.
0
 

Author Comment

by:Andrea_Corbo
ID: 39254954
This is a good idea, I will do this tests on friday and then I will update you.  This morning I spoke with a good Vmware technician  and he too was a bit 'surprised...

Bye bye
Andrea
0
 

Author Comment

by:Andrea_Corbo
ID: 39270772
hello,
Friday 'I could not go in the datacenter. Should I go tomorrow afternoon. I keep you updated.

Thank you.
0
 

Author Comment

by:Andrea_Corbo
ID: 39279095
Hello guys,
Today I tested the networking a lot, also with new switch and the issue between this two host persists.
Tomorrow I will reinstall Esxi5.1 on one host, than I'll do packet capturing like suggested by GILTJR.

bye bye
0
 

Author Comment

by:Andrea_Corbo
ID: 39289524
Hello everybody,
nothing has emerged from the recent tests done.
Yesterday I opened an incident in vmware.
when the issue will be resolved I will inform you.

thanks,
good day,
andrea
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39289859
Did the packet capture show a long delay anyplace?  Of course here "long" is relative, instead of 0.05 ms it might be 0.1ms.
0
 

Author Comment

by:Andrea_Corbo
ID: 39433697
Hi there!
the only thing vmware support found is that packet is divided whit a mtu of 60 . The problem however doesn't came from switch.  Next week I'm going to reinstall Hypervisor on that server and let's see...

I'll keep you informed.

bye bye Andrea

ps: I am very disappointed with the support received from vmware
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39433790
Using a MTU of 60 is going to cause some serious performance problems.

I would start looking at all Ethernet interface and see if you can find with with MTU or Ethernet framesize set real low.

Could be somebody meant to set it to 6000 and did a typo.
0
 

Author Comment

by:Andrea_Corbo
ID: 39433875
However we are going to setup esxi again, because we have lost too much time.

thank you all for the support....

closing the post

I hope the problem will go away!!!
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39434128
I hope it goes away too.  If it does not, then you need to start looking at other network equipment.

If the hosts are in the same ip subnet/vlan then it may be a ESXi issue.  If they are in different ip subnets/vlans then start looking at any/all routers/L3 devices.

Good Luck!
0
 

Expert Comment

by:Daniel J. Garcia
ID: 41821326
I am pretty sure that ESXi is limited in speed from the shell on purpose. When I use my own C program to copy data over a socket, speeds reaches 70-80 mb/s at sustained rates. After a few tries the speed starts to slow down until it gets stucked at 10 mb./s
0

Featured Post

Plug and play, no additional software required!

The ATEN UE3310 USB3.1 Gen1 Extender Cable allows users to extend the distance between the computer and USB devices up to 10 m (33 ft). The UE3310 is a high-quality, cost-effective solution for professional environments such as hospitals, factories and business facilities.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

During and after that shift to cloud, one area that still poses a struggle for many organizations is what to do with their department file shares.
What if you have to shut down the entire Citrix infrastructure for hardware maintenance, software upgrades or "the unknown"? I developed this plan for "the unknown" and hope that it helps you as well. This article explains how to properly shut down …
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…
Suggested Courses

615 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question