Solved

VMWare network configuration problems

Posted on 2011-09-23
8
1,770 Views
Last Modified: 2012-05-12
To start, this is my first VMWare setup ever.

I have purchased an IBM Bladecenter with 3 x HS22V VMWare servers and 1 HS22 Server 2008 R2 server.  The chassis has a pair of Cisco 3110G switches within that are stacked together.

I have, to the best of my knowledge and ability configured all three servers the exact same:

IP's 10.36.25.131 , 132 , and 133
Subnet Mask 255.255.255.192
Gateway 10.36.25.129

Vcenter server is 10.36.25.140

The Vcenter server is configured with LACP and the switch is configured accordingly and working.
The three ESXi 4.1.0 servers are set with both nics active and in the vsphere client they are all assigned to switch0 and configured with the "route based on IP Hash" setting in place.  The switch is setup with Etherchannel in place mode on.

Basically, I followed this document to fill in the gaps.
http://www.sysadmintutorials.com/tutorials/vmware-vsphere-4/vcenter4/network-teaming-with-cisco-etherchannel/

So...here is the problem:

Server 2 and 3 work perfectly, no issues.  
Server 1 however has a funny quirk where if I only enable NIC0, then only certain machines can ping the IP address, if I only enable NIC1 then other machines can ping the IP.

The Vcenter server can only contact VMWare 1 (131) when NIC0 is active only on the VMWare server.

There are multiple VLAN's on the network.  The servers are dedicated to VLAN 50 in this case.

Any suggestions.  The config of the switches is attached.  Thanks

Building configuration...

Current configuration : 4124 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname NSR-BLADECENTER-H
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$AseZ$pncXo.RtXU/Oyr2nZzjp3.
!
!
!
no aaa new-model
clock timezone EST -5
clock summer-time EDT recurring
switch 1 provision ws-cbs3110g-s-i
switch 2 provision ws-cbs3110g-s-i
system mtu routing 1500
authentication mac-move permit
ip subnet-zero
!
!
!
!
!
!
port-channel load-balance src-dst-ip
spanning-tree mode pvst
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
interface Port-channel1
 description **NIC TEAM VMWARE-A**
 switchport mode trunk
 spanning-tree portfast trunk
!
interface Port-channel2
 description **NIC TEAM VMWARE-B**
 switchport mode trunk
 spanning-tree portfast trunk
!
interface Port-channel3
 description **NIC TEAM VMWARE-C**
 switchport mode trunk
 spanning-tree portfast trunk
!
interface Port-channel14
 description **EC to VCenter LACP**
 switchport access vlan 50
 switchport mode access
!
interface Port-channel15
 description **EC BCSW to NSR3750**
 switchport mode trunk
!
interface FastEthernet0
 no ip address
 shutdown
!
interface GigabitEthernet1/0/1
 description **VMWARE-A**
 switchport mode trunk
 channel-group 1 mode on
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet1/0/2
 description **VMWARE-B**
 switchport mode trunk
 channel-group 2 mode on
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet1/0/3
 description **VMWARE-C**
 switchport mode trunk
 channel-group 3 mode on
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
!
interface GigabitEthernet1/0/14
 description **EC to VCENTER**
 switchport access vlan 50
 switchport mode access
 channel-group 14 mode active
 spanning-tree portfast
!
interface GigabitEthernet1/0/15
 description **UPLINK TO Network**
 switchport mode trunk
 channel-group 15 mode on
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet2/0/1
 description **VMWARE-A**
 switchport mode trunk
 channel-group 1 mode on
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/0/2
 description **VMWARE-B**
 switchport mode trunk
 channel-group 2 mode on
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/0/3
 description **VMWARE-C**
 switchport mode trunk
 channel-group 3 mode on
 spanning-tree portfast trunk
 spanning-tree bpdufilter enable
!
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
!
interface GigabitEthernet2/0/13
!
interface GigabitEthernet2/0/14
 description **EC to VCENTER**
 switchport access vlan 50
 switchport mode access
 channel-group 14 mode active
 spanning-tree portfast
!
interface GigabitEthernet2/0/15
 description **UPLINK TO Network**
 switchport mode trunk
 channel-group 15 mode on
!
interface GigabitEthernet2/0/16
!
interface GigabitEthernet2/0/17
!
interface GigabitEthernet2/0/18
!
interface Vlan1
 ip address 172.18.21.254 255.255.252.0
!
interface Vlan30
 ip address 10.36.25.5 255.255.255.128
!
ip default-gateway 10.36.25.1
ip classless
ip http server
!
ip sla enable reaction-alerts
!
line con 0
line vty 0 1
 password 7 011703085208071D25
 login
line vty 2 4
 login
line vty 5 15
 login
!
mac address-table static b8be.bf8f.831d vlan 1002 interface GigabitEthernet1/0/1
9
end

Open in new window

0
Comment
Question by:TheMetalicOne
8 Comments
 
LVL 5

Author Comment

by:TheMetalicOne
ID: 36589108
Do you think the attached image has anything to do with it?  What is the "Observed IP ranges"?
 NICs on VMWare1
0
 
LVL 9

Accepted Solution

by:
Bill_Fleury earned 250 total points
ID: 36589686
Observed IP ranges are any address ranges it's seen "out on the wire".

Having a difference in what's seen on each nic would indicate to me that you need to re-examine the VLAN configuration of the ports those nic's are connected to on the 3110G's, you will probably find a difference if you compare the config on those two ports.
0
 
LVL 35

Assisted Solution

by:Ernie Beek
Ernie Beek earned 250 total points
ID: 36589708
Observed ip range are literally that. The range that can be seen from the nic with the current settings. Are the settings for the two physical nics the same within the esx server? And did you assign the correct vlans to the virtual nics of the VMs?
0
 
LVL 35

Expert Comment

by:Ernie Beek
ID: 36589716
Hm, typing too slow again :(
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 5

Author Comment

by:TheMetalicOne
ID: 36590945
The config on the two switches in relation to the ports are the same.

In the config attached to the original post, the esxi server is tied to port-channel 1 which is the etherchannel for g1/0/1 and g2/0/1.
0
 
LVL 16

Expert Comment

by:danm66
ID: 36591251
what does the upstream switchport config look like?
0
 
LVL 5

Author Comment

by:TheMetalicOne
ID: 36598235
Here is the upstream switch.

The bladecenter switches are stacked, and the two servers I want to communicate between constantly are both in the bladecenter and on the same stacked set.
NSR-L3-STACK(config-if)#do sh run
Building configuration...

Current configuration : 2662 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname NSR-L3-STACK
!
boot-start-marker
boot-end-marker
!
enable secret 5 $1$wp93$BxvUsnOyILfp6CLqeEx5b.
!
no aaa new-model
switch 1 provision ws-c3750g-12s
switch 2 provision ws-c3750g-12s
system mtu routing 1500
ip subnet-zero
!
!
!
!
!
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
interface Port-channel1
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Port-channel5
 description **EC NL3->NL2**
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface Port-channel15
 description **EC to Bladcenter**
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet1/0/1
 description **EC to 2960S Stack**
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 5 mode on
!
interface GigabitEthernet1/0/2
 description **EC to Bladcenter**
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 15 mode on
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode on
!
interface GigabitEthernet2/0/1
 description **EC to 2960S Stack**
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 5 mode on
!
interface GigabitEthernet2/0/2
 description **EC to Bladcenter**
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 15 mode on
!
interface GigabitEthernet2/0/3
!
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode on
!
interface Vlan1
 ip address 172.18.21.193 255.255.252.0
 no ip route-cache
!
interface Vlan30
 ip address 10.36.25.3 255.255.255.128
 no ip route-cache
!
ip default-gateway 10.36.25.1
ip classless
ip http server
!
!
control-plane
!
!
line con 0
line vty 0 1
 password 7 0202015702050E3348
 login
line vty 2 4
 login
line vty 5 15
 login
!
ntp clock-period 36029008
ntp server 172.18.20.235
end

Open in new window

0
 
LVL 5

Author Closing Comment

by:TheMetalicOne
ID: 36598296
Ok, I just figured it out.  

Very simply put, on the vswitch, the configuration was correct to route by IP Hash, however in the vsphere client, the config of the management network was still set to route based on original IP Port.  

Once I changed it to IP Hash, everything started working.

Thanks
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

It Is not possible to enable LLDP in vSwitch(at least is not supported by VMware), so in this article we will enable this, and also go trough how to enabled CDP and how to get this information in vSwitches and also in vDS.
This is an issue that we can get adding / removing permissions in the vCSA 6.0. We can also have issues searching for users / groups in the AD (using your identify sources). This is how one of the ways to handle this issues and fix it.
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now