Solved

Cisco Nexus connectivity between 1G and 10G interfaces

Posted on 2014-02-11
17
1,778 Views
Last Modified: 2014-02-22
Hello,  I have two Cisco Nexus 5500 series switches that I am using as the core network for a new environment. I have a couple of physical servers connected to the Nexus via 1GB SFP modules. They are on vlan 5, subnet 10.35.5.x/24 and I have an Ision x200 NAS storage system (4 node cluster)  connected via their 10Gig Ethernet interfaces to the pair of Nexus via fiber optic patch cables. The storage interfaces are on vlan 10 , subnet 10.35.10.x/24.

The issue I have is connectivity between the 1Gig and 10Gig interfaces. I cannot ping any of the 10Gig interfaces (vlan 10) from the servers on the 1Gig interfaces (vlan 5). If I place a server on vlan 10, the server can ping and connect to the storage.  I know the config and routing are working correctly between the vlans. I have placed a server on each vlan and did not have any connectivity problems. The issue seems to involve the use of the 10Gig interfaces.

Here are my configurations:

Nexus 1

interface Vlan5
  no shutdown
  description "Servers"
  ip address 10.35.5.2/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    priority 120
    ip 10.35.5.1

interface Vlan10
  no shutdown
  description "Storage"
  ip address 10.35.10.2/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    priority 120
    ip 10.35.10.1


Nexus 2


interface Vlan5
  no shutdown
  description "Servers"
  ip address 10.35.5.3/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    ip 10.35.5.1

interface Vlan110
  no shutdown
  description "Storage"
  ip address 10.35.10.3/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    ip 10.35.10.1



All the storage interfaces are configured as follows:

interface Ethernet1/9 (thru 12)
    description "storage"
    switchport access vlan 110

All the server interfaces are configured as follows"

interface Ethernet1/16
  description "server"
  switchport access vlan 105
  speed 1000

The switches have a basic config right now since its still in development phase. I would greatly appreciate If anyone has any config ideas or troubleshooting tips.

Thank you.
0
Comment
Question by:ItSecurePro
  • 9
  • 7
17 Comments
 
LVL 12

Expert Comment

by:Infamus
ID: 39851006
Why do you have vlan 110 instead of vlan 10 on the Nexus 2?

Also why is the int gi1/16 access to 105, not vlan 5?

vlan numbers are not matching up.

And also interface Ethernet1/9 (thru 12) <----are they vpc?
0
 

Author Comment

by:ItSecurePro
ID: 39851087
My apologies, that was a type-o with the vlan number. The actual vlans are 105 and 110. there is no vlan error on the config. I pasted my config notes, not from the actual switches.

No, interfaces 9-12 are not configured as a vpc.
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39851124
are you able to ping vlan 110 interface (10.35.10.1) from server (vlan 105)?

And also, why the hsrp numbers are the same on vlan 105 and vlan 110 and don't have priority?
0
 

Author Comment

by:ItSecurePro
ID: 39851167
Yes, I can successfully ping vlan 110 interface 10.35.10.1 from a server on vlan 105.
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39851173
Sorry, I edited my comment.

How about HSRP settings?

Which IP are you using for the gateway on the server and the storage?
0
 

Author Comment

by:ItSecurePro
ID: 39851220
HSRP settings for Nexus 1 are set with priority of 120 and nothing is set for Nexus 2.

Server Gateway 10.35.5.1
Storage Gateway 10.35.10.1
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39851221
Try to set vpc priority on the second switch.

Also this is the vpc config


feature vpc

vpc domain 1
  role priority 1000
  peer-keepalive destination (IP address of the other Nexus Switch)
  delay restore 180
  peer-gateway
  auto-recovery

interface port-channel1
  description Storage
  switchport mode trunk
  switchport trunk native vlan xxx
vpc 1

interface Ethernet1/9 - 12
  description Storage_NIC#0-3
  switchport mode trunk
  switchport trunk native vlan xxx
  speed 10000
  channel-group 1 mode active
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39851226
Also try changing the gate way to vlan interface on Nexus 1 on the server and the storage and see if that works.
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 

Author Comment

by:ItSecurePro
ID: 39851634
Ok, I tried changing the gateway on the server and storage to the vlan interfaces (10.35.10.2 and 10.35.5.2). I lost connectivity to the devices.

I will create a port channel for the storage interfaces and then add them to a vpc like you suggested.
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39851664
lost connectivity?

Which switch is the server connected?
0
 

Author Comment

by:ItSecurePro
ID: 39851683
The second switch.  So I should of used gateway 10.35.5.3 for server. The storage is connected to the first switch, gateway 10.35.5.2. Trying that now.
0
 

Author Comment

by:ItSecurePro
ID: 39851719
With the gateway's changed I still cannot ping the storage from the server.
0
 
LVL 12

Expert Comment

by:Infamus
ID: 39851775
that's weird....

Can you actually post config for vlan 105, 110 and interface config for the server and storage?

And how's the storage network configured? Are the NIC's bonded?
0
 
LVL 50

Expert Comment

by:Don Johnston
ID: 39853479
There's something screwy here... I think it may be my interpretation of the problem. :-)

Let me see if I got this right:

Two devices on different VLANs can communicate as long as either one is not connected to a 10g interface? Is that correct?

Can two devices on the same (or different) VLANs communicate if they are both connected via 10g interfaces?
0
 

Author Comment

by:ItSecurePro
ID: 39853896
I am sorry for the late response.  I am testing connectivity scenarios again. I will reply to your questions shortly.
0
 

Accepted Solution

by:
ItSecurePro earned 0 total points
ID: 39865660
OK I finally got it! The issue was the port-channel config was mismatched with the Isilon x200 10Gig interface LACP config.

Isilon x200 10G intefaces:
Use LACP and aggregation mode (interface bonding) for the pair of 10G interfaces. Isilon node 1, 10G ext-1 is connected to Nexus 1 Port 9, 10G ext-2 is connected to Nexus 2 Port 9. They need to be configured in the same port-channel.

Nexus 1

interface Ethernet1/9
  switchport access vlan 110
  channel-group 901 mode active

Nexus 2
interface Ethernet1/9
  switchport access vlan 110
  channel-group 901 mode active

Then I created a vPC for this port-channel, vPC ID 901, Port Po901.

And so on with Isilon Node's 2,3, and 4. This corrected the connectivity issue. Now Servers on VLAN 105 and communicate with Storage on VLAN 110 and I can mount NFS shares.
0
 

Author Closing Comment

by:ItSecurePro
ID: 39879003
This networking config has been tested and verified.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

The worst thing when starting a new job is when the previous Network Administrator left behind no documentation. How do you get into the devices? If you've been in this situation or just accidently mistyped your password, this article will hopefully…
This tutorial will go through the steps required to write a script that will back up the configuration settings of a HP-ProCurve switch. You will need to get the following things to follow this tutorial: Telnet Scripting Tool e.g. TST10.exe …
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …
This tutorial demonstrates a quick way of adding group price to multiple Magento products.

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now