Solved

transfer rate between esxi hosts very slow

Posted on 2013-06-14
21
3,913 Views
Last Modified: 2016-11-23
Dear experts,
I have a strange problem with speed between two ESXi 5.1 hosts managed by a vCenter.
The machines involved are two Dell PowerEdge 2950 (local datastore, one has 15k disks, the other 7k2) , connected together by redundant gigabit network.

Each of them has a local datastore on which runs some VMs.

If I do a file transfer between this hosts by Datastore Browser or by a backup software (VMX emplorer), the speed that I reach is about 6/10 MB/sec.

You might think of a problem resident on the network, but performing a file transfer between VMs runnings on different host , I reach good performance (60/100 MB/sec). I also try to disable redundant nic for sake, but problem stiil remains.

We may think of datastore I/O problem of one of the two hosts, but it is not so because benchmarking each datastore, each result is good.

Any suggesitons?

thank's a lot

andrea
0
Comment
Question by:Andrea_Corbo
  • 11
  • 7
  • 2
  • +1
21 Comments
 
LVL 119
ID: 39247523
it's likely to be caused by the speed of reading and writing to VMFS partitions.
0
 

Author Comment

by:Andrea_Corbo
ID: 39247570
...may be...
I try to give you even this element: I also have a nfs shared storage (very cheap, qnap basi model).
From and to this nfs storage,  the speed reached by each ESXi hosts is the maximum possible for this nfs storage (30MB/sec).

So I remain confused...
0
 
LVL 119
ID: 39247616
are jumbo frames enabled on your VMKernel?

are the RAID and disk types the same?
0
What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

 

Author Comment

by:Andrea_Corbo
ID: 39247741
I just setted up MTU on VSwitch  at 9000 (also anabled in phisic HP switch).

NOW file transfer speed from server B to server A is back to over 100MB/sec, but if i do inverse operation, from SERVER B to A (copying the same file), speed is again slow, at 10MB/sec...

the raid is 6 type on both servers.

thank's a lot for your support
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 500 total points
ID: 39249376
How are you doing the file transfer?

Do a packet capture for just a few seconds in each direction.  Verify that jumbo frames are being use in both directions.  If the file transfer method uses TCP, verify the window size is the same, or close, in both directions.
0
 

Author Comment

by:Andrea_Corbo
ID: 39249384
errata corrige to the previous post: speed problem persists.

I have tried and tested for many hours (iperf, sqlio, etc.), but  any host to host transfers is slow  (vmotion, clone, vSphere replication, past / copy from the datastore browser) from 5/6MB to 15MB/sec.

On Monday I'm going to change / try new switches.

I will update you, thank's for now....
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39251553
What is the RTT if you ping the hosts from each other?
0
 

Author Comment

by:Andrea_Corbo
ID: 39251880
Ping from host 10.1.1.204 to 10.1.1.206 (dell server)

PING 10.1.1.206 (10.1.1.206): 56 data bytes
64 bytes from 10.1.1.206: icmp_seq=0 ttl=64 time=2.950 ms
64 bytes from 10.1.1.206: icmp_seq=1 ttl=64 time=0.620 ms
64 bytes from 10.1.1.206: icmp_seq=2 ttl=64 time=0.371 ms

consider that between this hosts is now running a backup job and vmware replication....
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39251936
So no obvious errors on the pings.  A packet capture, just a few seconds in each direction, may show what is going on.

What is the CPU utilization like?  If the CPU's are being maxed out, it will affect network transfer rates.
0
 

Author Comment

by:Andrea_Corbo
ID: 39252393
Cpus and RAM in both server are very low. About packet capture, is there something embedded on esxi host console or do I mirror eth port on my switch and get data?

thanks
0
 
LVL 57

Accepted Solution

by:
giltjr earned 500 total points
ID: 39254435
I beleive that tcpdump should exist on esxi.

You can capture the traffic and write it to a file, then transfer the file to your local computer and use Wireshark to look at the capture.

Something like:

tcpdump -s 0 -i xxxxx -w file01.cap

The character that follows the -s is the number zero.

where xxxxx is the name of the interface you want to capture on.
0
 

Author Comment

by:Andrea_Corbo
ID: 39254954
This is a good idea, I will do this tests on friday and then I will update you.  This morning I spoke with a good Vmware technician  and he too was a bit 'surprised...

Bye bye
Andrea
0
 

Author Comment

by:Andrea_Corbo
ID: 39270772
hello,
Friday 'I could not go in the datacenter. Should I go tomorrow afternoon. I keep you updated.

Thank you.
0
 

Author Comment

by:Andrea_Corbo
ID: 39279095
Hello guys,
Today I tested the networking a lot, also with new switch and the issue between this two host persists.
Tomorrow I will reinstall Esxi5.1 on one host, than I'll do packet capturing like suggested by GILTJR.

bye bye
0
 

Author Comment

by:Andrea_Corbo
ID: 39289524
Hello everybody,
nothing has emerged from the recent tests done.
Yesterday I opened an incident in vmware.
when the issue will be resolved I will inform you.

thanks,
good day,
andrea
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39289859
Did the packet capture show a long delay anyplace?  Of course here "long" is relative, instead of 0.05 ms it might be 0.1ms.
0
 

Author Comment

by:Andrea_Corbo
ID: 39433697
Hi there!
the only thing vmware support found is that packet is divided whit a mtu of 60 . The problem however doesn't came from switch.  Next week I'm going to reinstall Hypervisor on that server and let's see...

I'll keep you informed.

bye bye Andrea

ps: I am very disappointed with the support received from vmware
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39433790
Using a MTU of 60 is going to cause some serious performance problems.

I would start looking at all Ethernet interface and see if you can find with with MTU or Ethernet framesize set real low.

Could be somebody meant to set it to 6000 and did a typo.
0
 

Author Comment

by:Andrea_Corbo
ID: 39433875
However we are going to setup esxi again, because we have lost too much time.

thank you all for the support....

closing the post

I hope the problem will go away!!!
0
 
LVL 57

Expert Comment

by:giltjr
ID: 39434128
I hope it goes away too.  If it does not, then you need to start looking at other network equipment.

If the hosts are in the same ip subnet/vlan then it may be a ESXi issue.  If they are in different ip subnets/vlans then start looking at any/all routers/L3 devices.

Good Luck!
0
 

Expert Comment

by:Daniel J. Garcia
ID: 41821326
I am pretty sure that ESXi is limited in speed from the shell on purpose. When I use my own C program to copy data over a socket, speeds reaches 70-80 mb/s at sustained rates. After a few tries the speed starts to slow down until it gets stucked at 10 mb./s
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Short answer to this question: there is no effective WiFi manager in iOS devices as seen in Windows WiFi or Macbook OSx WiFi management, but this article will try and provide some amicable solutions to better suite your needs.
This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question