ESX 3.5 network delays

we have an ESX 3.5/update 5 server with 1Gbit network and SCSI U320 hdds.

when copying files between vitrual machines and a neighbour servers (they are in the same 1Gb network segment) we encounter a speed limit of ~100Mb instead of 1G.

- The ifconfig on ESX host shows 1G connected, and the same speed is on VM's network card
- no collisions detected on a hosted network card
- no traffic shaper is applied to a VM's virtual switch
- network card is a "VMXNET"
- no problems was found with IRQ sharing
- VM's HDD is a "RAW-type"
- no limits on shared resources
- disk shares is set to "HIGH"
Y0sh1Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

za_mkhIT ManagerCommented:
When you say a speed limit of 100mb ...  do you mean it is transfering at 100 Megabytes a second or a 100 Megabits a second?
How do you know there is speed limitation?
Apologies in advance if these are stupid questions ...
0
bbnp2006Commented:
vmware tools installed?
0
Y0sh1Author Commented:
The speed is about 7-9 Megabytes per\sec which is near 100Mbits/sec networking

vmware tools installed and is up to date
0
Protecting & Securing Your Critical Data

Considering 93 percent of companies file for bankruptcy within 12 months of a disaster that blocked access to their data for 10 days or more, planning for the worst is just smart business. Learn how Acronis Backup integrates security at every stage

bbnp2006Commented:
what OS is your VM running?
0
Y0sh1Author Commented:
windows server 2003 x64
0
bbnp2006Commented:
VMXNET 2 (enhanced) or VMXNET 3?
0
Y0sh1Author Commented:
Enhanced vmxnet. Also tried on E1000 - no differences where found in delays. So i thing with vmxnet 3 there will be no diffs also.
0
bbnp2006Commented:
all our windows 2003 server VMs are using VMXNET 3 type vNIC and it works perfectly.
0
Y0sh1Author Commented:
when i added a vNIC there where only two choices of E1000 and Enhanced vmxnet.  No VMXNET3 where suggested. To add VMXNET3 i followed these steps:
1.Upgraded VMware tools (from VMWare infrastructure console i'v choosed upgrade tools by right-click on VM)
2. Shutdown VM
3. Tryed to find an 'upgrade hardware' by right clicking with no success.

The version of VM is 4, while VMXNET3 wants in to be at least  7.
The version of Vmware infrastructure client i use is  2.5.0 build  62276.  Seems like i need to upgrade the client to create a version 7 VMs
0
bbnp2006Commented:
Sounds about right :) VMXNET 3 is available for VMs on hardware ver7
0
Y0sh1Author Commented:
well, i'v upgraded client to 2.5.0 build 204931, but it still creates version 4 VMs, no upgrade hardware is available by right click.
0
vmwarun - ArunCommented:
Set the speed 1000 Mbps and duplex to Full manually instead of Auto config on both the ESX Host and physical switch port and check the transfer speeds.
0
Y0sh1Author Commented:
setting the speed 1000 Mbps and duplex to Full manually not helped
0
vmwarun - ArunCommented:
I suggest that you check your network cables which interconnect ESX Hosts and physical switches. (CAT 5 would provide 100 Mbps while CAT 5E or CAT 6 would provide 1000 Mbps)
0
Y0sh1Author Commented:
I'll check the cable tomorrow, but if we'd have a CAT 5,  the link status wouldnt be a 1000G/Full ,right?
0
Paul SolovyovskySenior IT AdvisorCommented:
Also check to see if there are switches in the between the two segments that are 100mbps
0
Y0sh1Author Commented:
well, just checked - cable is CAT 5E and there is no 100Mb switches in between.
0
Y0sh1Author Commented:
have to add that when networking files between VMs (hosted on the same ESX host) the speed is about 20Mbytes/sec
0
Paul SolovyovskySenior IT AdvisorCommented:
have you tried to put a physical system on the same source and destination switch, perhaps a port setting or uplink?  How many nics do you have on the vswitch and how is it setup?
0
Y0sh1Author Commented:
ESX host and other servers are on the same 1G switch with no shapers  applied on its ports. The link status on both ends is 1G/Full

There is only one NIC (Broadcom NetXtreme II BCM5708) on the vswitch. with 1G/Full (manual)

There is another NIC (BCM5708) that is dedicated to a service console.
0
vmwarun - ArunCommented:
When you transfer files between two virtual machines that are a part of the same port group in a vSwitch, then the traffic does not go out and is routed within the vSwitch so transfer speed would be higher.
Try changing the cable to a different CAT5E or CAT 6 and check.
0
Y0sh1Author Commented:
have changed the cable to a different CAT5 E - not helped.
0
kugaconsultantsCommented:
When you say copying files from VMs to other servers, you're talking about copying to physical servers?  How about copying files from physical servers to VMs?  Do you still have the same slow copy performance?

I'm thinking that the write cache on the server's HDDs or controllers you're copying to is not sufficient to handle higher transfers through the network.  Also, you may want to check the processor on both servers copying because remember TCP/IP is not offloaded to the NIC unless you have TOE enabled or an HBA.  Your server's processor may be running near capacity which will be the reason why trasnfers speeds are not optimal.
0
Y0sh1Author Commented:
I am talking about copying files between VMs and other physical servers that are locating on the same physical 1Gbit switch.
When copying files from VM to Physical server the speed is 31Megabytes/sec
When copying files from Physical server to VM the speed is about 5Megabytes/sec

No matter where files are copying to - whether it be a local ESX host storage or a NAS disk attached via lun.
CPU utilization is at 17 percents at maximum

i'll check the write buffers later
0
Y0sh1Author Commented:
i have to notice (again) that copying files between VMs have no any delays and the speed is 30Megs/sec. So i can conclude that nothing is wrong with the storage write cache. It is something else...
0
Y0sh1Author Commented:
when copying files via FTP from external server into COS's partition, the delays also persist. I think it has to be something with tuning of the  host physical networking. possible iptables or other stuff...maybe some unnoticed default ESX network shaper...dont know yet
0
Paul SolovyovskySenior IT AdvisorCommented:
What is the setting on the vswitch that is going out to your production network, how are you doing it? MAC, IP Hash, etc...?  Are you running trunking on your vswitch?
0
Y0sh1Author Commented:
On vswitch i have only one physical NIC.  NIC teaming is configured by default and the settings are:
Load Balancing: Route based on the originating virtual port ID
Network failover detection: Link status only
Notify switches: yes
Failback: Yes

Traffic shaping is disabled
Security is set to Accept all

Configured Speed, Duplex: 1000Mb, Full Duplex

On the "virtual machine network" properties the VLAN ID is set to "none", so no trunkin is configured
0
Vijay kumar MohanrajCloud ArchitechCommented:
If your VM is only reporting 100Mbps network speed, that is because you are using the VLANCE VMware NIC instead of the VMXNET NIC. Don't worry, it's a false indicator, your virtual NIC is still transmitting packets as fast as possible based on the PNIC or system bus speed, but you should upgrade to the VMXNET NIC wherever and whenever possible as it is a more efficient and slightly faster NIC than the slower-more-compatible VLANCE which is a virtualized old AMD PCNET Ethernet standby.
0
Y0sh1Author Commented:
(referring to my 1st post)  all NICs are "VMXNET"
0
Y0sh1Author Commented:
It was FC HBA's that is causing delays. The problem was solved by Updating the HBA's BIOS to the latest version.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
vmwarun - ArunCommented:
Its nice to see that your problem got resolved.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.