?
Solved

ESXi network speed theoretical max between two virtual guests on the same platform

Posted on 2010-01-12
13
Medium Priority
?
3,339 Views
Last Modified: 2012-06-27
If we put two virtual machines on the same ESXi server host within the same network and virtual switch (vSwitch), should the communication be greater than "wire speed" (Gig E in this case) or will it be metered down to simulate Gig E. I assume it could it be even slower as depends on the cpu in as much as the networking is cpu-bound, but  what is the theoretical max?

I have in mind, specifically, two linux guests on the 192.168.1.x transfering files to one another.
0
Comment
Question by:rzup
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 4
  • 2
  • +1
13 Comments
 
LVL 57

Accepted Solution

by:
giltjr earned 252 total points
ID: 26295563
Actually the slow point could be CPU or it could be (more than likely) DASD I/O rates.

Assuming that you are not running this on a severely under powered physical server, you will get over Gig E.  I would setup for jumbo frames as jumbo frames will reduce the CPU overhead on each of the VM's.

For ESI 3.5 you can read:

http://www.vmware.com/files/pdf/ESX_networking_performance.pdf

For Windows VM to Windows VM it was up to 1.6 Gbps

For Linux VM to Linux VM it was up to 2.6 Gbps.
0
 
LVL 1

Author Comment

by:rzup
ID: 26297120
Excellent article, thanks, I will try switching to vmnet drivers and try jumbo frames for SuSE 64 systems. My virtual guests have slower than expected performance, sometimes as if they must "wake up" and I am trying to track it down.
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 252 total points
ID: 26297918
Now, I have no clue what  your environment is, but there are a few reasons that why that might be.

1) You do not have enough physical RAM in the ESXi host to support the virtual hosts you have active.
2) You do not have the physical CPU's in the ESXi host to support the virtual hosts you have active.
3) You are transferring small files (either a few or many, does not matter as long as they are "small".

TCP has a slow start function to help prevent flooding the network with data that it can't handle.  Some times you have to transfer 10's of MB's before TCP can figure out that it has a LOT of bandwidth with very little latecy.

So a 1 MB file will not get the same through put as a 100 MB  file, in fact one-hundred 1 MB files will take longer to send than one 100 MB file.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 1

Author Comment

by:rzup
ID: 26298078
Sometimes when you connect to the virtual machines they respond sluggishly in the shell (the echo is slow), then they speed up after a minute or so. As if under a load that quickly goes away--but there is no significant load or load average. I have not seen this with native machines of the same function, OS and configuration. Hence my comment that they seem momentarily "asleep".

Often, and somewhat contradictory to the above, when we start transferring a file with, say, scp, the speed will begin at 30Mbps and then steadily drop to below a 4Mbps. I have tried very large files and watched the throughput steadily drop. Again, I do not see this on native machines.

We have 32gigs of ram on a brand new 4-cpu 24 core r905 Dell, with just a few virtual machines on it. Also the resource meter doesn't suggest the sluggishness should be present. Hence, I'm guessing the sometimes erratic cpu and network performance are some soft of configuration issue or combination of configuration issues--that seems like a powerful box. to struggle with only four to five lightly loaded servers. We use it for light QA duty, one box has SQL Server and Windows Server 2003 on it--again lightly used by QA.

The setup is very new. I'm trying to get grounded and better document the issues. I'm going to start with that netperf benchmark software mentioned in the article you sent and make sure I'm getting the symptoms straight. It was helpful to see the article on native versus virtual to virtual performance. Thanks.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 26299258
How much total RAM do you have allocated to the virtual hosts?

When transfers start off at a high rate and then drop off that typically indicates that there is a network bottle neck, but since the network is a virtual switch, it could also mean that the receiving OS is having a problem writing the file out to the disk due to I/O contention.

Although I will admit that 4 Mbps a second would me a LOT of I/O contention.  
0
 
LVL 1

Author Comment

by:rzup
ID: 26299391
The most virtual ram I configured is 8gigs (middleware, db) and frontends have 4gigs, which works well in actual production. The sum total of the virtual machines is actually below the total of physical memory (db=8, mw1=8, mw2=8, fe=4 : 8+8+8+4 < 32)

Disk I/O could be a possibility, but I have a RAID 5 consisting of 5 15k SAS internal disks on the r905--not exactly the slowest of disk systems all things considered.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 26299637
Is it 30 mega bits per second, or 30 mega bytes?

Does it go down to 4 mega bits or mega bytes?

Although even at mega bytes it is much, much slower that I would expect, if it is truly mega bits, then it way slower that what it should be.
0
 
LVL 1

Author Comment

by:rzup
ID: 26300773
megabytes MB, sorry
0
 
LVL 24

Assisted Solution

by:ryder0707
ryder0707 earned 124 total points
ID: 26301602
Looking at you Q, I think in order to properly evaluate VM network performance is to transfer data from VM to another physical host in the network not to another VM in the same host
If you copy/move files within between 2 VMs in the same host, logically you are limiting the transfer speed to read/write speed of physical disk spindle

Network performance will be different for each below networking setup as well

1) 2 VMs on different vSwitch, same port group and vlan
2) 2 VMs on same vSwitch, different port group and vlan
3) 2 VMs in the same vSwitch, same port group and vlan

If you decide to reevaluate your test, perhaps you should change the network adapter to vmxnet3, this is the best performance virtual nic under vSphere refer to http://www.vmware.com/pdf/vsp_4_vmxnet3_perf.pdf
0
 
LVL 21

Assisted Solution

by:za_mkh
za_mkh earned 124 total points
ID: 26307708
I am a bit late coming to this question but I want to address the authors observation of 'waking up' the VMs. I experience this with lots of our servers (windows). I find this only happens the the VM in question is not being used i.e. there is no major I/o going through it and it is sitting idle.
So if there is no load in the VM, the ESX hosts seems to 'clear out' the memory / etc so that I guess it can use it for other VMs. This is how I like to to think it works. When there is activity, the ESX host then furiously winds the VM up again to give it the processing / memory  power that it needs ... normally takes about 20-60 seconds, and then the VM is as responsive as ever. I say this because I am normally the first person in the office in the morning and when I log into some systems, I have observed this, but by the time other people decide to use the system, everything is as good as apple pie!
I don't think you need to be overly concerned by that behaviour, to be honest, but with regards to file copy transfer issues, I don't know. We definitely don't experience that.
0
 
LVL 1

Author Comment

by:rzup
ID: 26309124
za_mkh, that is a very interesting observation on initial latency. Your the first to suggest that this may be normal. Thanks.
0
 
LVL 1

Author Comment

by:rzup
ID: 26309138
ryder0707,

Thanks, given we are using 15k SAS spindles, I would tend to think that vm to vm should still be faster than what we are seeing. I am retesting tomorrow with the netperf software to make sure I am getting accurate numbers.

Thanks again.
0
 
LVL 24

Expert Comment

by:ryder0707
ID: 26309830
yeah thats is what I'm trying to explain, ofcoz transfer files in the same host is expected to be faster, well a lot faster then network speed, but try copying huge files between partition on the same disk and compare the speed when transferring files between partition on different disk
i'm just saying the test setup is not really the proper way to evaluate network performance
if you want to make it a real "network" test then i guest you should involve physical networking not virtual, so pNIC to pNIC
Look at the pdf file from vmware and observe the test bed setup
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If we need to check who deleted a Virtual Machine from our vCenter. Looking this task in logs can be painful and spend lot of time, so the best way to check this is in the vCenter DB. Just connect to vCenter DB(default DB should be VCDB and using…
Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
Teach the user how to rename, unmount, delete and upgrade VMFS datastores. Open vSphere Web Client: Rename VMFS and NFS datastores: Upgrade VMFS-3 volume to VMFS-5: Unmount VMFS datastore: Delete a VMFS datastore:
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question