Solved

How to properly configure Loadbalance and other settings in Portgroup with 4 NICs coming off 2 different core switches

Posted on 2013-01-16
22
653 Views
Last Modified: 2013-01-29
I have recently setup new vSphere5.1 environment comprising of 3 ESX Host DL580G7(32Cores-384GBRAM)

I have also been asked to add 4 NICs to my "Production" portgroup that will host all Servers

I personally wanted 2 but had to settle down for 4 coz of line manager demand

I need some help to properly configure loadbalancing options to best utilise 4 dedicated Physical NICs to this port group

I tried to use "Etherchannel" but vmware only supports etherchannel coming off same switch. This way i was going to lose my resilience across 2 core switches

So currently i have 2 NICs connected to a Cisco 4506 and 2 NICs connected to Cisco 6509 switch

Please advice best way to configure loadbalancing settings on portgroup

Find attached screenshot of settings that i need changed
0
Comment
Question by:sysprof
  • 9
  • 8
  • 3
22 Comments
 

Author Comment

by:sysprof
ID: 38783112
0
 
LVL 118

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 38783558
On the Cisco configuration you will need to setup an Etherchannel Trunk for Two NICs on the Cisco 4506 and  setup an Etherchannel Trunk for Two NICs on the  Cisco 6509.

On the ESX server vSwitch configuration you will want to set it to Route based on ip hash

Sample configuration of EtherChannel / Link aggregation with ESXi/ESX and Cisco/HP switches

also see here

http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/
0
 

Author Comment

by:sysprof
ID: 38792254
As per article
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2034277

"vSphere supports only one LACP group per distributed switch and only one LACP group per host"

It would mean i cant have an etherchannel coming off 4506 and another coming off6509 in same port group. I have already given up on this one due to this reason

Unless you interpret it differently?

Im actually trying to undertand how to best utilise/configure 4 NICs that are connected to my "Produciton" Port group and do some VMWARE load balancing
0
 
LVL 118
ID: 38792286
we usually configured ESXi servers, with trunks to Cisco switches, two ports to one Cisco switch, and two ports to another Cisco switch.

running VLANs for all portgroups.

the Cisco switches we use are 4509s if I recall. We used to use 6509

I can dig out the Cisco config off the switches, if it's of any help.
0
 

Author Comment

by:sysprof
ID: 38792608
It's strange coz article suggests we can't have 2 ether channels coming off 2 different switches Do u interpret article differently ? It will be good to see config and also what was show etherchannel load balance output?
0
 
LVL 118
ID: 38792712
I'll get the configs off the switches.
0
 
LVL 118
ID: 38818972
Okay, reminder from the Mod!

We actually use Cisco 4506s, we moved all our servers off Cisco 6509s.

this is the working config at our Datacentre.

CYRUS_DC_YRK_1_4500_1#sho etherchannel load-balance
EtherChannel Load-Balancing Operational State (src-dst-ip):
Non-IP: Source XOR Destination MAC address
  IPv4: Source XOR Destination IP address
  IPv6: Source XOR Destination IP address
CYRUS_DC_YRK_1_4500_1#

Open in new window


CYRUS_DC_YRK_1_4500_1 Config

interface GigabitEthernet2/8
 description ESX004
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 5,7,8,703,705
 switchport mode trunk
 speed 1000
 duplex full
 spanning-tree portfast trunk

interface GigabitEthernet6/19
 description ESX004
 switchport access vlan 5
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 5,7,8,703,705
 switchport mode trunk
 speed 1000
 duplex full
 spanning-tree portfast trunk
 spanning-tree bpduguard enable

Open in new window


CYRUS_DC_YRK_1_4500_2 Config

interface GigabitEthernet2/8
 description ESX004
 switchport access vlan 5
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 5,7,8,703,705
 switchport mode trunk
 speed 1000
 duplex full
 spanning-tree portfast trunk

interface GigabitEthernet6/19
 description ESX004
 switchport access vlan 5
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 5,7,8,703,705
 switchport mode trunk
 speed 1000
 duplex full
 spanning-tree portfast
 spanning-tree bpduguard enable

Open in new window

0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 250 total points
ID: 38819099
hanccocka:

Unless I am missing something I don't see where you are using Etherchannels in your config. they are trunks, but not Etherchannels.  I don't see any channel-group definitions.  Which means as far as the 4500's are concerned these are 4 totally independent connections.

Not sure what this means for the VMWare side.   I know for "standard Linux" we NIC team two NIC's and put each NIC in a different switch and have no real problem.  This does mean that only one NIC is used for inbound traffic, while both NIC's can be used for outbound traffic.
0
 
LVL 118
ID: 38819168
as detailed in http:#a38792286 above, probably my terminology between trunks and etherchannel, and mixing the technologies.

I'll search the rest of the config
0
 

Author Comment

by:sysprof
ID: 38819187
in my current configurations there are no etherchannels either. I have 2 Gig ports each coming from 4506 & 6509 to my port group "Production" on vsphere5.1. Thus making it 4 Physical NICs that are part of port group "Production"

I want to know how should i best configure the "Port Group" settings

I sent screenshot early but re-attaching it again to explain where i need help with

I have spoken to VMWare and they tell me same as mentioned in article (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2034277)

that 2 etherchannels coming off 2 different switches cant be part of same Port Group so im not worrying about etherchannels anymore
Production-PortGroup-with-4NICS.GIF
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 118
ID: 38819209
we have two trunks, one trunk from  one switch, one trunk from another switch to esx servers.

using above config on the switches.

as for port group settings

Route based on IP Hash load balancing
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 250 total points
ID: 38819259
I checked and we use Route based on IP Hash load balancing also.

For the guests VMWare will attempt to load the guests across the 4 NIC's, however a single guests traffic will all go over the same physical path while there is active TCP connections.  Don't ask how UDP works.  This is because a single IP address can only be associated with a single MAC in a arp table at any point in time.

If you have a active TCP connection going across a NIC and that NIC fails, VMWare should send out a proxy ARP to change the associated MAC.
0
 

Author Comment

by:sysprof
ID: 38826012
thx-

I have changed "loadbalancing" to "IP Hash" See attached

How about other options like "Network Failover Detection" or "Notify Switches"?

Also any recommended good test so i can make sure that im best utilising my 4 different physical Nics coming off 2 Core switches.

I personally wanted to use 2 instead of 4 but team decided to go for 4

Would you personally recommend 2 or 4? I feel this has been little over engineered
PortGroupSettings.JPG
0
 
LVL 118

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE)
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
ID: 38826257
it very much depends on how much traffic your VMs are generating, and whether your links are saturated.

check Cisco counters on the physical switch, use esxtop in network mode

One of the biggest management holes in vCenter of ESX is the vSphere Client can indicate that VM network traffic is causing a 1 GB Ethernet adapter to have a 99% utilization rate. But strangely, it doesn't display which kind of traffic is going across the virtual networks, where it came from or where it's going.

To learn which traffic is going across a virtual network, there's another free tool for vSphere: Xangati for ESX, a virtual appliance that tracks conversations on the virtual network. It's great for troubleshooting any virtual network issue, analyzing virtual desktop infrastructure and correlating vCenter performance stats with virtual network stats.

and then you can really check what traffic from your VMs for free is causing network issues.

It's available as a fanastic FREE download here.

http://xangati.com/try-it-free/
0
 

Author Comment

by:sysprof
ID: 38826810
Could someone also comment on my question wrt "loadbalancing" settings

i attached screenshot earlier
0
 

Author Comment

by:sysprof
ID: 38826818
BTW, I have only selected load balancing option "IP hash" but i got this event logged on my core switch where my host is directly connected

2 NICs connected to same host, had port gi4/9 and gi6/40 flapping this morning

See below:
*Jan 28 06:53:52.602: %C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:A2:3E:2F in vlan 50 is flapping between port Gi4/19 and port Gi6
/40

Any ideas how to avoid this?
0
 
LVL 118
ID: 38826918
set  Link Staus only for failover detection.

Okay flapping is normal, because ESX server is sending from both nics.

you can create an etherchannel trunk, and this warning will stop.

 
interface Port-channel2

description VSH03 Etherchannel Group 1

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10-13,30

switchport mode trunk

switchport nonegotiate

!

interface Port-channel3

description VSH03 Etherchannel Group 2

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 10-13,30

switchport mode trunk

switchport nonegotiate

!

interface Port-channel4

description VSH03 Etherchannel V-Motion

switchport access vlan 50

switchport mode access

 

 

interface GigabitEthernet3/19

description VSH03 - Nic 2

switchport access vlan 50

switchport mode access

channel-group 4 mode on

!

interface GigabitEthernet3/18

desc VSH03 - Nic 3

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

channel-group 3 mode on

!

interface GigabitEthernet2/46

desc VSH03 - Nic 4

switchport trunk encapsulation dot1q

switchport mode trunk

switchport nonegotiate

channel-group 2 mode on

Open in new window

0
 

Author Comment

by:sysprof
ID: 38826942
ok,

How about other options like "Network Failover Detection" or "Notify Switches"? Should i configure or leave them greyed out?
0
 
LVL 57

Expert Comment

by:giltjr
ID: 38826966
hanccocka is correct about creating Etherchannel on the switches.

However, please remember since you are connecting to two different switches you could still have some issues.

You have configured VMWare to group 4 NIC's together as 1, but they are going to two different switches that do NOT support "cross switch Etherchannels."  

Basically what you are trying to do with the best of intentions is not really supposed to be done.

I will have to double check with our VMWare team because I thought we were doing this, but I know we had problems in the beginning so it is possible that they changed the config on the VMWare side and never told us.
0
 
LVL 118
ID: 38826982
leave them off.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
Veeam Backup & Replication has added a new integration – Veeam Backup for Microsoft Office 365.  In this blog, we will discuss how you can benefit from Office 365 email backup with the Veeam’s new product and try to shed some light on the needs and …
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now