Link to home
Start Free TrialLog in
Avatar of o-tvw-ee
o-tvw-ee

asked on

ESX on IBM Bladecenter VLAN problem

Hi all,

We have bought an IBM BladeCenter H with 4 blades. Each blades has 2 NIC's and each NIC is connected to one of two Bladecenter compatible Nortel GB Switches.
Everything is working as long as we don't use VLAN's.

For our OOB network we've created a seperate portgroup (VLAN 101) and we want to connect this to an untagged VLAN net work of our external Dell PowerConnect 6248 switch.

On the 2 Nortel switches we also defined a VLAN 101 on all internal ports (tagged) and on 1 external ports (untagged) and have connected the external port to the vlan on a DELL PowerConnect 6248 (also untagged).

This is not working. We have a similar setup with a Dell PowerEdge 2950 (no blade) and that's working just fine.

I assume something is wrong in the intermediate Nortel switches (I don't have much experience with Nortel) since both the Dell Switch and the ESX are configured in the same way that the working setup is.
In attachment I have a few overviews of the setup.

What am I doing wrong?

Thanks!

esx.png
vlan1.png
vlan2.png
vlan3.png
Avatar of Rick_O_Shay
Rick_O_Shay
Flag of United States of America image

The Nortel switches use the PVID in conjunction with the VLAN tag. Try setting that to autopvid. I am not familiar with that user interface. What model Nortel switches are these.
Avatar of o-tvw-ee
o-tvw-ee

ASKER

They are custom made for IBM Bladecenters: Nortel Layer2-3 GbE Switch Module(Copper).


This is our current configuration file:

Switch is in I/O Module Bay 1
 
Current idle CLI timeout: 5 minutes
Use of BOOTP for configuration currently disabled
Display hostname (sysName) in CLI prompt currently disabled
Syslog console enabled
Port-based Port Mirroring currently disabled
IP Forwarding currently ON
NTP currently ON
Primary NTP server address 83.101.44.113
NTP resync interval 1440 minutes
Daylight savings time: OFF
RADIUS authentication currently OFF
TACACS+ authentication currently OFF
LDAP Authentication currently OFF
Current SNMP params:
  Read community string:  "public"
  Write community string: "private"
  Trap source address:  0.0.0.0
  Authentication traps disabled.
  All link up/down traps enabled.
 
Current v1/v2 access enabled
 
Current SNMPv3 USM user settings:
  1: name adminmd5, auth md5, privacy des
  2: name adminsha, auth sha, privacy des
  3: name v1v2only, auth none, privacy none
 
Current SNMPv3 vacmAccess settings:
  1: group name admingrp, model usm
     level authPriv,
     read view iso, write view iso, notify view iso
  2: group name v1v2grp, model snmpv1
     level noAuthNoPriv,
     read view iso, write view iso, notify view v1v2only
 
Current SNMPv3 vacmSecurityToGroup settings:
  1: model usm, user name adminmd5, group name admingrp
  2: model usm, user name adminsha, group name admingrp
  3: model snmpv1, user name v1v2only, group name v1v2grp
 
Current SNMPv3 vacmViewTreeFamily settings:
  1: name v1v2only, subtree 1
     type included
  2: name v1v2only, subtree 1.3.6.1.6.3.15
     type excluded
  3: name v1v2only, subtree 1.3.6.1.6.3.16
     type excluded
  4: name v1v2only, subtree 1.3.6.1.6.3.18
     type excluded
  5: name iso, subtree 1
     type included
 
Current System Access settings:
IP Management currently allowed from *ALL* IP addresses
 
 
Usernames:
  user     - enabled    - offline
  oper     - disabled   - offline
  admin    - Always Enabled - online     2 sessions.
Current User ID table:
Current strong password settings:
  strong password status: disabled
 
HTTP access currently enabled on TCP port 80
HTTPS server access currently disabled on TCP port 443
SNMP access currently read-write
User configuration from BBI currently disabled
TFTP occurs over port 69.
 
 
port     - Port Menu
------------------------------------------------------------------
Current port INT1 configuration: enabled, PVID 1, VLAN tagged
    name INT1
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT2 configuration: enabled, PVID 1, VLAN tagged
    name INT2
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT3 configuration: enabled, PVID 1, VLAN tagged
    name INT3
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
Current port INT4 configuration: enabled, PVID 1, VLAN tagged
    name INT4
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT5 configuration: enabled, PVID 1, VLAN tagged
    name INT5
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT6 configuration: enabled, PVID 1, VLAN tagged
    name INT6
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT7 configuration: enabled, PVID 1, VLAN tagged
    name INT7
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
Current port INT8 configuration: enabled, PVID 1, VLAN tagged
    name INT8
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT9 configuration: enabled, PVID 1, VLAN tagged
    name INT9
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT10 configuration: enabled, PVID 1, VLAN tagged
    name INT10
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT11 configuration: enabled, PVID 1, VLAN tagged
    name INT11
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT12 configuration: enabled, PVID 1, VLAN tagged
    name INT12
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT13 configuration: enabled, PVID 1, VLAN tagged
    name INT13
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port INT14 configuration: enabled, PVID 1, VLAN tagged
    name INT14
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
Current port MGT1 configuration: enabled, PVID 4095 tagged, VLAN tagged
    name MGT1
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port MGT2 configuration: enabled, PVID 4095 tagged, VLAN tagged
    name MGT2
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port EXT1 configuration: enabled, PVID 1
    name EXT1
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port EXT2 configuration: enabled, PVID 101
    name EXT2
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port EXT3 configuration: enabled, PVID 102
    name EXT3
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port EXT4 configuration: enabled, PVID 1 tagged, VLAN tagged
    name EXT4
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port EXT5 configuration: enabled, PVID 1
    name EXT5
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
Current port EXT6 configuration: enabled, PVID 1
    name EXT6
    802.1p priority: 0
    DSCP remarking for port: disabled
    Fast forwarding mode: disabled
ACL Port config is empty
 
 
l2       - Layer 2 Menu
------------------------------------------------------------------
 
upfast disabled, update 40
 
bpdu guard disabled
 
MAC address notification disabled
 
Spanning Tree Groups:
------------------------------------------------------------------
Current Spanning Tree Group 1 settings: ON
 
Bridge params:  Priority  Hello  MaxAge  FwdDel  Aging
                 32768      2      20      15     300
VLANs:  1 101 102 200
STP Ports:
Port INT1 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT2 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT3 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT4 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT5 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT6 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT7 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT8 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT9 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT10:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT11:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT12:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT13:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT14:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port EXT1 :  Priority 128, Path Cost 0 auto
Port EXT2 :  Priority 128, Path Cost 0 auto
Port EXT3 :  Priority 128, Path Cost 0 auto
Port EXT4 :  Priority 128, Path Cost 0 auto
Port EXT5 :  Priority 128, Path Cost 0 auto
Port EXT6 :  Priority 128, Path Cost 0 auto
------------------------------------------------------------------
Current Spanning Tree Group 128 settings: ON
 
Bridge params:  Priority  Hello  MaxAge  FwdDel  Aging
                 32768      2      20      15     300
VLANs:  4095
STP Ports:
Port INT1 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT2 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT3 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT4 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT5 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT6 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT7 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT8 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT9 :  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT10:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT11:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT12:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT13:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port INT14:  Priority 128, Path Cost 0 auto, edge, Spanning Tree turned OFF
Port MGT1 :  Priority 128, Path Cost 4 auto, Spanning Tree turned OFF
Port MGT2 :  Priority 128, Path Cost 4 auto, Spanning Tree turned OFF
 
All Trunk groups are disabled.
 
VLANs:
Current VLAN 1:
    name "Default VLAN", ports INT1-INT14 EXT1 EXT4-EXT6, enabled, management disabled, bandwidth disabled,
    Protocol- empty,
    spanning tree 1
Current VLAN 4095:
    name "Mgmt VLAN", ports INT1-INT14 MGT1 MGT2, enabled, management enabled, bandwidth disabled,
    Protocol- empty,
    spanning tree 128
Current VLAN 101:
    name "OOB", ports INT1-INT14 EXT2, enabled, management disabled, bandwidth disabled,
    Protocol- empty,
    spanning tree 1
Current VLAN 102:
    name "DMZ", ports INT1-INT14 EXT3, enabled, management disabled, bandwidth disabled,
    Protocol- empty,
    spanning tree 1
Current VLAN 200:
    name "CUSTOMER", ports INT1-INT14 EXT4, enabled, management disabled, bandwidth disabled,
    Protocol- empty,
    spanning tree 1
 
802.1x:
System status : disabled
Guest VLAN status : disabled
Guest VLAN : 101
 
                    Quiet     Tx    Max   Supp    Server   ReAuth  ReAuth
Port   Auth Mode    Period  Period  Req  Timeout  Timeout  Status  Period
----- ------------  ------  ------  ---  -------  -------  ------  ------
  G   force-auth        60      30    2      30       30     off     3600
INT1  force-auth        60      30    2      30       30     off     3600
INT2  force-auth        60      30    2      30       30     off     3600
INT3  force-auth        60      30    2      30       30     off     3600
INT4  force-auth        60      30    2      30       30     off     3600
INT5  force-auth        60      30    2      30       30     off     3600
INT6  force-auth        60      30    2      30       30     off     3600
INT7  force-auth        60      30    2      30       30     off     3600
INT8  force-auth        60      30    2      30       30     off     3600
INT9  force-auth        60      30    2      30       30     off     3600
INT10 force-auth        60      30    2      30       30     off     3600
INT11 force-auth        60      30    2      30       30     off     3600
INT12 force-auth        60      30    2      30       30     off     3600
INT13 force-auth        60      30    2      30       30     off     3600
INT14 force-auth        60      30    2      30       30     off     3600
MGT1  force-auth        60      30    2      30       30     off     3600
MGT2  force-auth        60      30    2      30       30     off     3600
EXT1  force-auth        60      30    2      30       30     off     3600
EXT2  force-auth        60      30    2      30       30     off     3600
EXT3  force-auth        60      30    2      30       30     off     3600
EXT4  force-auth        60      30    2      30       30     off     3600
EXT5  force-auth        60      30    2      30       30     off     3600
EXT6  force-auth        60      30    2      30       30     off     3600
-------------------------------------------------------------------------------
G - Global port configuration
 
 
l3       - Layer 3 Menu
------------------------------------------------------------------
Current IP configuration:
  rearp 10, gw metric strict
  router ID 0.0.0.0, AS number 0
 
Current interfaces:
128: 192.168.40.111  255.255.248.0   192.168.47.255,  vlan 4095, enabled
 
Current default gateways:
132: 192.168.40.2,    intr 2, retry 8, arp enabled, vlan 4095, enabled
 
Current static routes:
ECMP health-check ping interval: 1
ECMP health-check retries number: 3
ECMP Hash Mechanism: sip
 
Current IP forwarding settings: ON, dirbr disabled, noicmprd disabled
 
 
Current IP port settings:
INT1 : ON
INT2 : ON
INT3 : ON
INT4 : ON
INT5 : ON
INT6 : ON
INT7 : ON
INT8 : ON
INT9 : ON
INT10: ON
INT11: ON
INT12: ON
INT13: ON
INT14: ON
MGT1 : ON
MGT2 : ON
EXT1 : ON
EXT2 : ON
EXT3 : ON
EXT4 : ON
EXT5 : ON
EXT6 : ON
 
Current network filter settings:
  none
 
 
Current route map settings:
 
Current RIP settings: OFF, update 30
 
Current OSPF settings: OFF
  Default route none
  lsdb limit 0
 
Current BGP settings:
  OFF, pref 100, AS number 0
 
Current BGP peer settings:
 
Current BGP aggr settings:
 
 
Current DNS settings:
  0.0.0.0, 0.0.0.0, none
 
Current BOOTP relay settings: OFF
  0.0.0.0, 0.0.0.0
 
Current ARP configuration:
  rearp 10
No static ARP configured.
 
 
qos      - QOS Menu
------------------------------------------------------------------
 
 
8021p    - 802.1p Menu
------------------------------------------------------------------
Current priority to COS queue configuration:
Number of COSq: 2
Priority  COSq  Weight
--------  ----  ------
    0       0      1
    1       0      1
    2       0      1
    3       0      1
    4       1      2
    5       1      2
    6       1      2
    7       1      2
 
 
dscp     - Remark DSCP value to a new DSCP value
------------------------------------------------------------------
Current DSCP Remarking Configuration: OFF
 
  DSCP    New DSCP  New 802.1p Prio
--------  --------  ---------------
    0         0          0
    1         1          0
    2         2          0
    3         3          0
    4         4          0
    5         5          0
    6         6          0
    7         7          0
    8         8          1
    9         9          0
   10        10          1
   11        11          0
   12        12          1
   13        13          0
   14        14          1
   15        15          0
   16        16          2
   17        17          0
   18        18          2
   19        19          0
   20        20          2
   21        21          0
   22        22          2
   23        23          0
   24        24          3
   25        25          0
   26        26          3
   27        27          0
   28        28          3
   29        29          0
   30        30          3
   31        31          0
   32        32          4
   33        33          0
   34        34          4
   35        35          0
   36        36          4
   37        37          0
   38        38          4
   39        39          0
   40        40          5
   41        41          0
   42        42          0
   43        43          0
   44        44          0
   45        45          0
   46        46          5
   47        47          0
   48        48          6
   49        49          0
   50        50          0
   51        51          0
   52        52          0
   53        53          0
   54        54          0
   55        55          0
   56        56          7
   57        57          0
   58        58          0
   59        59          0
   60        60          0
   61        61          0
   62        62          0
   63        63          0
 
 
acl      - Access Control List Menu
------------------------------------------------------------------
  No ACLs configured.
  No ACL blocks configured.
  No ACL groups configured.
 
 
pmirr    - Port Mirroring Menu
------------------------------------------------------------------
Port Mirroring is disabled
 
Monitoring port Mirrored ports
INT1            none
INT2            none
INT3            none
INT4            none
INT5            none
INT6            none
INT7            none
INT8            none
INT9            none
INT10           none
INT11           none
INT12           none
INT13           none
INT14           none
MGT1            none
MGT2            none
EXT1            none
EXT2            none
EXT3            none
EXT4            none
EXT5            none
EXT6            none

Open in new window

I don't see where it is in this config but make sure you are not set for dropping untagged or unregistered frames on ports in VLAN 101. As I mentioned before also make sure autopvid is on.
As far as I know STP has to be disabled on the Nortel ports connecting to ESX. Our Nortel 5510e configuration guide for VMware ESX specifically said that STP had to be disabled - but maybe this only for MLT ports created to ESX servers.
And this could be me talking complete rubbish:
Also do you mean to imply that ports connected to your dell power connect are part of VLAN 101 and as such are untagged? Not knowing how Dell Switches are configured, but if it was set that all ports are automatically part of VLAN 101 then maybe your uplink from the Nortel to the Dell also has to be tagged for the packets to get across? Becuase the way I read it, a packet is tagged to be part of VLAN 101 when it leaves the the ESX server up to the Nortel Switch. But when it leaves the Nortel Switch on it its way to the Dell Switch stack it does that on an untagged port, which would strip out VLAN 101. But then where was it supposed to go?
We have several untagged VLAN's on our Dell switch (so actually, just a few switches in one hardware box). now we want to connect some of these 'switches' to our ESX servers.
The traffic between the vSwitch and Nortel is tagged, but the Nortel switch should untag them for a certain port so the traffic can go untagged to the dell switch.

I'll check both your suggestions and come back to you!
Well, one more point.
So if I assume that you have three untagged switches in your dell box each hosting an 'untagged' VLAN, I can assume you are using each swtich to have three 'physically' seperate LANS hence you saying they are untagged.
To enable traffic to pass between these LANS you are using routing or or dedicated switch ports to send traffic between the 'LANS.
Hopefully my assumption is correct.
If the assumption is correct,  I then think you need to configure the Dell port connecting to your Nortel switches to strip out the VLAN tag? I know you said the dell port is configured as untagged -but I think it should be told what to do if it does get a packet with a VLAN IDentifier in it.
Hope this helps.
We don't allow inter-vlan traffic, but for the rest your assumption is right. The Dell switch is configured to send untagged packets to the Nortel switch. This is working when we connect another ESX server directly to the Dell server.

The problem is just that the Nortel switch currently is not removing the VLAN tag. But I'll have to look into the suggestions made before.
When you are using virtual switch VLAN tagging (that is, your VLANs are set at the vSwitch or port group level in ESX), your physical switch ports that connect to the ESX NICs should be in trunk mode. This is because they need to handle any number of VLANs you configure in your ESX.
Alternatively, if you only ever want to use that one VLAN on your ESX, remove the VLAN configuration from your OOB port group and leave it configured at the physical switch level.
 
 
To FilipZahradnik,
You are indeed correct on the need to configure trunks, but one doesnt have to exactly configure trunking for the physical switch for VLANs to work. This of course depends on how a person's network is physically setup e.g you could configure your vSwtich portgroup to be part of VLAN101 e.g. but its uplink could be to a switch that only handles traffic for VLAN101 anyway, so need to make that uplink port a trunk.
But for the most case, trunks are advised!  
From the author's description, it appears that he has VLAN 101 configured on both the port group in ESX (author's first screenshot) and the physical switch ports that connect to the ESX server (INT ports in the second screenshot).
If I may speculate, but the issue could potentially be caused by how the Nortel switches handle this double tagging (maybe the Dell switch handles it differently and therefore that setup works).
Hi all,

we've got some more time to investigate this.
We started from a new situation. We have the ESX servers with a tagged VLAN 200 going to one of the NIC's. that NIC is connected to the built-in Nortel switch. If we create an external port on the Nortel switch in VLAN 200, untagged, and connect a laptop to it, the VLAN works just fine.

If we add another external port to VLAN 200 and make it a tagged switch, connect this switch to our DELL switch (that port is also tagged for VLAN 200) we get some problems. We are unable to send any traffic over VLAN 200 to/from the Dell switch/Cisco FW. The LAN (VLAN 1) works just fine.

If we take a look at the Nortel switch we see that STP is blocking the port (EXT4) that is connected to the VLAN on the DELL switch. So the switch sees a loop somewhere (I think).. But we can't find it...

I've attached a visio drawing of the current situation (the Dell switch has some more ports tagged for VLAN 200, but those are working fine, the problem is somewhere between the Dell switch and the Nortel switch) and a screenshot of the STP information on the Nortel Switch.

Any ideas?
(I couldn't find the autopvid information)
overview.jpg
nortel.png
If you want to reduce the situation further, maybe you can try only working with one of the BladeCenter Nortel switches to see if the loop goes away.
Also, in your screenshots, it appears that red VLAN200 links are connected to ports 3 and 5 on your Dell switch. Ports 3 and 5 have different configuration (3 is marked T and 5 is marked U). Should the 2 ports be configured identically?
Spanning tree is blocking because there is a loop due to the multiple ports in both switches connecting  to each other. Spanning tree is run amongst all of the ports in the spanning tree group not per VLAN. All of the VLANS mentioned are in SPT group 1. This is fine but what you need to do is configure the multiple links between the switches into one trunk called MLT on the Nortel switch and add all of the VLANs that need to go between both switches to the trunk.
The bladecenter is connected to ports 3 and 4 of the Dell switch.Sorry, my bad.

We also tested with only 1 Nortel switch connected, but we got the same error.
If we disconnect the LAN interface on the Nortel switch the problem goes away...

I'll try Rick_O_Shay's suggestion.
ASKER CERTIFIED SOLUTION
Avatar of za_mkh
za_mkh
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial