Cisco Nexus connectivity between 1G and 10G interfaces

Hello,  I have two Cisco Nexus 5500 series switches that I am using as the core network for a new environment. I have a couple of physical servers connected to the Nexus via 1GB SFP modules. They are on vlan 5, subnet 10.35.5.x/24 and I have an Ision x200 NAS storage system (4 node cluster)  connected via their 10Gig Ethernet interfaces to the pair of Nexus via fiber optic patch cables. The storage interfaces are on vlan 10 , subnet 10.35.10.x/24.

The issue I have is connectivity between the 1Gig and 10Gig interfaces. I cannot ping any of the 10Gig interfaces (vlan 10) from the servers on the 1Gig interfaces (vlan 5). If I place a server on vlan 10, the server can ping and connect to the storage.  I know the config and routing are working correctly between the vlans. I have placed a server on each vlan and did not have any connectivity problems. The issue seems to involve the use of the 10Gig interfaces.

Here are my configurations:

Nexus 1

interface Vlan5
  no shutdown
  description "Servers"
  ip address 10.35.5.2/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    priority 120
    ip 10.35.5.1

interface Vlan10
  no shutdown
  description "Storage"
  ip address 10.35.10.2/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    priority 120
    ip 10.35.10.1


Nexus 2


interface Vlan5
  no shutdown
  description "Servers"
  ip address 10.35.5.3/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    ip 10.35.5.1

interface Vlan110
  no shutdown
  description "Storage"
  ip address 10.35.10.3/24
  ip router eigrp 100
  ip passive-interface eigrp 100
  hsrp version 2
  hsrp 101
    preempt delay minimum 120
    ip 10.35.10.1



All the storage interfaces are configured as follows:

interface Ethernet1/9 (thru 12)
    description "storage"
    switchport access vlan 110

All the server interfaces are configured as follows"

interface Ethernet1/16
  description "server"
  switchport access vlan 105
  speed 1000

The switches have a basic config right now since its still in development phase. I would greatly appreciate If anyone has any config ideas or troubleshooting tips.

Thank you.
ItSecureProAsked:
Who is Participating?
 
ItSecureProConnect With a Mentor Author Commented:
OK I finally got it! The issue was the port-channel config was mismatched with the Isilon x200 10Gig interface LACP config.

Isilon x200 10G intefaces:
Use LACP and aggregation mode (interface bonding) for the pair of 10G interfaces. Isilon node 1, 10G ext-1 is connected to Nexus 1 Port 9, 10G ext-2 is connected to Nexus 2 Port 9. They need to be configured in the same port-channel.

Nexus 1

interface Ethernet1/9
  switchport access vlan 110
  channel-group 901 mode active

Nexus 2
interface Ethernet1/9
  switchport access vlan 110
  channel-group 901 mode active

Then I created a vPC for this port-channel, vPC ID 901, Port Po901.

And so on with Isilon Node's 2,3, and 4. This corrected the connectivity issue. Now Servers on VLAN 105 and communicate with Storage on VLAN 110 and I can mount NFS shares.
0
 
InfamusCommented:
Why do you have vlan 110 instead of vlan 10 on the Nexus 2?

Also why is the int gi1/16 access to 105, not vlan 5?

vlan numbers are not matching up.

And also interface Ethernet1/9 (thru 12) <----are they vpc?
0
 
ItSecureProAuthor Commented:
My apologies, that was a type-o with the vlan number. The actual vlans are 105 and 110. there is no vlan error on the config. I pasted my config notes, not from the actual switches.

No, interfaces 9-12 are not configured as a vpc.
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

 
InfamusCommented:
are you able to ping vlan 110 interface (10.35.10.1) from server (vlan 105)?

And also, why the hsrp numbers are the same on vlan 105 and vlan 110 and don't have priority?
0
 
ItSecureProAuthor Commented:
Yes, I can successfully ping vlan 110 interface 10.35.10.1 from a server on vlan 105.
0
 
InfamusCommented:
Sorry, I edited my comment.

How about HSRP settings?

Which IP are you using for the gateway on the server and the storage?
0
 
ItSecureProAuthor Commented:
HSRP settings for Nexus 1 are set with priority of 120 and nothing is set for Nexus 2.

Server Gateway 10.35.5.1
Storage Gateway 10.35.10.1
0
 
InfamusCommented:
Try to set vpc priority on the second switch.

Also this is the vpc config


feature vpc

vpc domain 1
  role priority 1000
  peer-keepalive destination (IP address of the other Nexus Switch)
  delay restore 180
  peer-gateway
  auto-recovery

interface port-channel1
  description Storage
  switchport mode trunk
  switchport trunk native vlan xxx
vpc 1

interface Ethernet1/9 - 12
  description Storage_NIC#0-3
  switchport mode trunk
  switchport trunk native vlan xxx
  speed 10000
  channel-group 1 mode active
0
 
InfamusCommented:
Also try changing the gate way to vlan interface on Nexus 1 on the server and the storage and see if that works.
0
 
ItSecureProAuthor Commented:
Ok, I tried changing the gateway on the server and storage to the vlan interfaces (10.35.10.2 and 10.35.5.2). I lost connectivity to the devices.

I will create a port channel for the storage interfaces and then add them to a vpc like you suggested.
0
 
InfamusCommented:
lost connectivity?

Which switch is the server connected?
0
 
ItSecureProAuthor Commented:
The second switch.  So I should of used gateway 10.35.5.3 for server. The storage is connected to the first switch, gateway 10.35.5.2. Trying that now.
0
 
ItSecureProAuthor Commented:
With the gateway's changed I still cannot ping the storage from the server.
0
 
InfamusCommented:
that's weird....

Can you actually post config for vlan 105, 110 and interface config for the server and storage?

And how's the storage network configured? Are the NIC's bonded?
0
 
Don JohnstonInstructorCommented:
There's something screwy here... I think it may be my interpretation of the problem. :-)

Let me see if I got this right:

Two devices on different VLANs can communicate as long as either one is not connected to a 10g interface? Is that correct?

Can two devices on the same (or different) VLANs communicate if they are both connected via 10g interfaces?
0
 
ItSecureProAuthor Commented:
I am sorry for the late response.  I am testing connectivity scenarios again. I will reply to your questions shortly.
0
 
ItSecureProAuthor Commented:
This networking config has been tested and verified.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.