Cisco UCS Fiber Channel Port Channel work around on mds 6120xp switches

Casey HermanCitrix Engineer
Published:
The 6120xp switches seem to have a bug when you create a fiber port channel when you have a UCS fabric interconnects talking to them.  If you follow the Cisco guide for the UCS, the FC Port channel will never come up and it will say that there are no members available. So, after a week of beating my head against the wall I came up with a method and so far has worked every time.  I think the main issue is the ports never reset their port state until you specifically disable the ports and then bring them back up again.  So here are my steps as follows.


If the instruction starts with UCS then you need to do the step on the UCS, if it starts with MDS then you need to do it on the fiber channel switches.

1) Install Gbics and get the physical connectivity going.
2) UCS Config the new ports as FC uplink ports (May already be done).
3) Open Cisco Device Manager on log into the appropriate switch.
4) MDS (Device Manager)  click on interface then portchannel.
5) MDS Click Create.
6) MDS increment the ID by one from last portchannel if not done already.
7) MDS set "nontrunk, auto, force, portvsan # depending on the switch, then hit the ... to select interfaces that are in the port channel.
8) UCS create port channel do not add ports then disable it then set the appropriate vsan.
9) UCS add ports.
10) MDS while holding control.. Select all ports in the portchannel then right click one of the selected ports and hit config.
11) MDS in the popup window set the status admin on the ports to down then hit apply.
12) UCS enable port channel.
13) MDS set status admin to up on the ports in the popup window and hit apply. In a about 2 minutes the port channel should come up.

Open your favorite SSH client and connect to the MDS switch that you are working with.  Log in with your username and password. Then type "term mon" to see the connection state while you are doing these steps. It will show in real time as you bring the ports up and down and show you the errors that are related to why the ports are working.  The console will refresh faster than the device manager GUI.

So once you have gone through these steps you can see where the ports have come up. Once you go through these steps you can take the port channel up and down as needed. It seems that the initial build out of the port channel is the issue and the ports on the MDS never go completely down and come back up without doing these steps.
0
10,170 Views
Casey HermanCitrix Engineer

Comments (2)

great article !

I am currently attempting to connect a cisco fabic interconnect to a MDS switch and then to an emc clarion CX 4480 via FC only .

to do this I am thinking that the first step is to set up the interconnect and then in your  step to is to configure the interconnect FC ports as FC uplink ports ,

I am very new to this and could use any advice you can give!
Casey HermanCitrix Engineer

Author

Commented:
Are you using multiple FI's and MDS's for Redundancy?  What model MDS I heard other models do not have this problem.  

Keep in mind (from what I have read and understand, feel free to correct me) that the failover for the FC is done via the server only.  In my example what I did is created a FC portchannel on each FI.  I have 2 vSan's 100 and 200. My MDS switch A carries all the 100 traffic and B carries all the 200 traffic.  So when I configure my UCS Portchannels FI A is set to vSAN 100 and portchannel on FI B is set to vSAN 200 and connected to their respective MDS switch.

So my 2 Gbics are 4 gig a piece in the portchannel. My sans have 2 adapters both 4 gig. Both sans are connected to both MDS switches for redundant paths.

Word of advice.. Build a server template with all vNic's vHBA's and everything you need. It will save some of the confusion later.

Create your MAC pools for your vNics and vHBA's.  So you can call one HBA_FabA and HBA_FABB. When it asks your for the naming scheme change one of the digits to A on the A side and B on the B side.  It makes it a lot easier to make sure your connections are connecting on the right vSan's / Adapters and vNics and what not.  

If I go into vmware I can look at the MAC for the HBA and can tell what vSAN it is on or if it was mapped incorrectly in the Service Profile that was deployed for that server.

The Ethernet port channel in version 1.43 of the UCS manager had issues as well. The only way I got that to work was to create it command line. Once I update to 2.01(t) all was right with the world. Make sure you set the vNics to their respective FI and enable failover. Did a failover test with a VM running on VMware and it worked like a champ. I was pinging it and only lost one ping. Also had a short hiccup on the server I was logged into while it was failing over. Both for the Uplink Ethernet Portchannel and the FC portchannel.

A tip for performance I have found in vmware is to manage the Datastore Paths and set them to Round Robin. Else it will only use one path and use the secondary as a failover only. I didnt want that. I wanted all links to be utilized equally. The UCS/FI's/MDS's all seemed to handle that well. I migrated (vStorageMotion) 33gig of data from one SAN to the other in under 3 minutes. I was quite pleased.

Getting started on the UCS is quite overwhelming at first. Eventually it all makes sense though.  Not to mention that nothing is really real except the actual physical connections. Even the memory is virtualized to the CPU.

I am still learning everything myself but I think I have it almost completely figured out.

Storage.pdf

See the above for my diagram.

Good luck!

Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.