Solved

Trunking Ports to ESXi 5.5

Posted on 2014-01-19
25
329 Views
Last Modified: 2015-06-23
Hi, I'm configuring several ESXi servers and have several 10Gb ports on the host servers.

I have setup a trunk for each host server on my core 5406 switch.

The virtual machines that will run on the hosts will only run in VLAN10.

So far on the core switch I have tagged the trunks with VLAN10, is this right?

If it is, do I then just tag the vSwitch with VLAN10 as well, or leave it untagged?

5406-Core# sh ru

Running configuration:

; J8697A Configuration Editor; Created on release #K.15.12.0012
; Ver #05:08.41.ff.3f.ef:63
hostname "5406-Core"
module 1 type j9538a
module 2 type j9538a
module 3 type j9546a
module 4 type j9546a
module 5 type j9550a
trunk C8,D1 trk1 trunk
trunk C7,D2 trk2 trunk
ip route 0.0.0.0 0.0.0.0 10.1.100.10
ip routing
interface A1
   name "Uplink to Admin1"
   exit
interface A2
   name "Uplink to Admin2"
   exit
interface A3
   name "Uplink to Admin3"
   exit
interface B7
   name "Uplink to Admin4"
   exit
interface B8
   name "Uplink to Admin5"
   exit
interface C7
   name "VMHost 2 Data"
   untagged vlan 1
   trunk trk2 trunk
   exit
interface C8
   name "VMHost 1 Data"
   untagged vlan 1
   trunk trk1 trunk
   exit
interface D1
   name "VMHost 1 Data"
   untagged vlan 1
   trunk trk1 trunk
   exit
interface D2
   name "VMHost 2 Data"
   untagged vlan 1
   trunk trk2 trunk
   exit
interface E1
   name "Uplink to Firewall"
   exit
snmp-server community "public" unrestricted
vlan 1
   name "DEFAULT_VLAN"
   no untagged A1-A3,B7-B8,E1-E13
   untagged A4-A8,B1-B6,C1-C6,D3-D8,E14-E24,Trk1-Trk2
   ip address dhcp-bootp
   exit
vlan 10
   name "Servers"
   untagged E2-E13
   tagged Trk1-Trk2
   ip address 10.1.10.1 255.255.255.0
   exit
vlan 15
   name "iSCSI_A"
   ip address 10.1.15.1 255.255.255.0
   exit
vlan 16
   name "iSCSI_B"
   ip address 10.1.16.1 255.255.255.0
   exit
vlan 17
   name "vMotion"
   ip address 10.1.17.1 255.255.255.0
   exit
vlan 18
   name "vHA"
   ip address 10.1.18.1 255.255.255.0
   exit
vlan 20
   name "Workstations"
   tagged A1-A3,B7-B8
   ip address 10.1.20.1 255.255.255.0
   ip helper-address 10.1.10.10
   ip helper-address 10.1.10.11
   exit
vlan 30
   name "Printers"
   tagged A1-A3,B7-B8
   ip address 10.1.30.1 255.255.255.0
   ip helper-address 10.1.10.10
   ip helper-address 10.1.10.11
   exit
vlan 40
   name "Private_WiFi"
   tagged A1-A3,B7-B8
   ip address 10.1.40.1 255.255.255.0
   ip helper-address 10.1.10.10
   ip helper-address 10.1.10.11
   exit
vlan 41
   name "Warehouse_WiFi"
   tagged A1-A3,B7-B8
   ip address 10.1.41.1 255.255.255.0
   ip helper-address 10.1.10.10
   ip helper-address 10.1.10.11
   exit
vlan 42
   name "Public_WiFi"
   tagged A1-A3,B7-B8
   ip address 10.1.42.1 255.255.255.0
   ip helper-address 10.1.10.10
   ip helper-address 10.1.10.11
   exit
vlan 99
   name "Switch_Management"
   tagged A1-A3,B7-B8
   ip address 10.1.99.1 255.255.255.0
   exit
vlan 100
   name "Firewall_In"
   untagged E1
   ip address 10.1.100.1 255.255.255.0
   exit
spanning-tree
spanning-tree Trk1 priority 4
spanning-tree Trk2 priority 4
spanning-tree priority 0 force-version rstp-operation
no tftp server
no autorun
no dhcp config-file-update
no dhcp image-file-update
password manager

5406-Core#

Open in new window

0
Comment
Question by:Tech Man
  • 12
  • 10
  • 3
25 Comments
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 39792500
You must also add the Tag Number 10, to the virtual portgroup you create for your virtual machines, connected to the vSwitch, so traffic leaving the vSwitch, will be 802.1Q Tagged with the VLAN number 10.

Your physical switch configuration/switch will then underastand this traffic because it has an 802.1Q tag,

I did not see any VLAN 10 Tag configuration in the configuiration you uploaded for your trunks?
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39792516
Is this what your expecting to see?

interface Trk1
   tagged vlan 10
   spanning-tree priority 4
   exit
interface Trk2
   tagged vlan 10
   spanning-tree priority 4
   exit

Open in new window

0
 
LVL 117
ID: 39792538
That's better and correct

Also do not forget config to virtual port group
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39792738
Thanks, I'll test this tomorrow.

Should my iSCSI and vMotion VLANS have interface ip addresses?
0
 
LVL 117
ID: 39792773
iSCSI and vMotion are portgroups and need an IP address to carry traffic so this is correct.
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39793673
Ok, I've setup all of this and it appears to be working.

One question though. I take it anything I define in the Default Policies of the vSwitch, these will also apply to the port groups defined below it unless I specify otherwise?

vm Switch
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39797294
Ok, I've got this all setup with the config detailed above.

But when I start testing what would happen if there was a nic/port/cable failure I lose all connectivity to the esxi management network.

So if I pull a cable out for a minute, then patch it back it, the interface comes back up, but the esxi host acts like both network connections are down.

It then also reports that one nic is disconnected, even though the switch reports that is up.

If I disable the ports via the switch and re-enable them to simulate failure, they operate as expected and rarely miss a ping.

This is a log from the procurve 5406zl which is appears to be behaving as expected.

I 01/21/14 14:02:55 00076 ports: port C5 in Trk2 is now on-line
I 01/21/14 14:02:55 00435 ports: port C5 is Blocked by STP
I 01/21/14 14:02:16 00077 ports: port C5 in Trk2 is now off-line
I 01/21/14 13:58:02 00076 ports: port C6 in Trk2 is now on-line
I 01/21/14 13:58:02 00435 ports: port C6 is Blocked by STP
I 01/21/14 13:57:12 00077 ports: port C6 in Trk2 is now off-line
I 01/21/14 13:53:05 00076 ports: port C6 in Trk2 is now on-line
I 01/21/14 13:53:05 00435 ports: port C6 is Blocked by STP
I 01/21/14 13:52:30 00077 ports: port C6 in Trk2 is now off-line

Open in new window



To recover the esxi host I have to reset the networking.
But after that, I can only see one active nic.
Its like VMware has shutdown one of the nics.

Any help would be appreciated.

I'm just trying to trunk 10Gb ports from a Procurve 5406 to a ESXi server.
One 10Gb onboard nic and one 10Gb PCIe nic.

533FLR-T and 530T cards
0
 
LVL 117
ID: 39797508
Are you both trunked, on the physical nics?

if one nics goes down, it should carry on, on the different nic?
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39797557
both trunked and configured as per screen shots.

It works fine until you replicate a cable fault. e.g. pull the cable out for 1min.

I feel like STP on either the switch or the vswitch is causing it.

I'm at the point of not using trunks and going with one nic and a standby

esxi_host2_managementesxi_host2_vmnetworkesxi_host2_vswitch
interface Trk2
   tagged vlan 10
   spanning-tree priority 4
   exit


interface C5
   untagged vlan 1
   trunk trk2 trunk
   exit
interface C6
   untagged vlan 1
   trunk trk2 trunk
   exit

Open in new window

0
 
LVL 117
ID: 39797569
what teaming policy are you using?
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39797582
On VMware it is IP Hash, with both nics active
On the 5406zl its just trunk
0
 
LVL 117
ID: 39797605
see this article..

http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

this is how it should be configured your port channel....
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 1

Author Comment

by:Tech Man
ID: 39797727
I've read that, but still no clearer.
0
 
LVL 117
ID: 39797732
Is your physical switch config the same as per document?
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39797863
My physical switch config is as above, as recommended by VMware technotes
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39798509
I may be mistake but I thought is you are using LACP or Spanning Tree you should not use IP Hash as your NIC Teaming policy but use "Route based on the originating port ID"  instead...
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39804674
I'm not using LACP as I'm not using vDS, just standard vSwitches.

I'm using just a standard HP trunk trk1 trunk command.
0
 
LVL 117
ID: 39804712
did you look at your teaming policy?
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39805122
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39827841
In discussions with HP Network support, they claim that you would need to reset the VMware network for which the trunk is connected if it flapped.

Not ideal and not the solution I'm looking for.

I now not using trunks, just multiple connections to the switch and leaving VMware networking in its default configuration.

I works for me at present.
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39827893
I have seen issues like this at past where I work with flapping and it was caused with a mismatch in the VMware NIC teaming policy and switch settings? I know I am beating a death hose here but do be it sounds like an issue with you NIC teaming policy. Have you tried to use Route based on originating IP as the teaming policy as a test?
0
 
LVL 117
ID: 39827943
if you are using HP Networking.

1. LACP is not supported for Standard vSwitches.
2. Standard Trunks are required.
3. and the only teaming policy that supports that is Route based on IP Hash.
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39827971
The HP ports are setup as standard trunks.
The VMware vSwitches are setup as Route based on IP Hash

Recreating a network port flap e.g. pulling a network cable, waiting a minute, then reconnecting the cable causes the network on that trunk to go down completely.
The only way to recover it is to reset the networking on the vSwitch and reconfigure it.
0
 
LVL 117
ID: 39827993
ah, what other options are ticked?

We had this issue with a client recently, all was working correctly, until they tested, by pulling cables, and it all went very weird and stopped working....

Make sure also the following are selected:-

Load Balancing - Route Based on IP HASH
Network Failover detection - Link Status Only
Notify Switches - Yes
Rolling Failover - No

and both nics must be active.

and make sure that's the same ON ALL Management Network, vSwitch, VM Network so it's consistent.
0
 
LVL 1

Author Comment

by:Tech Man
ID: 39903283
I gave up on this in the end.
I stopped using trunks and just used esxi in the standard setup.

it appears to function as I expect.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now