Cisco UCS Fiber Channel Port Channel work around on mds 6120xp switches

AID: 9633
  • Status: Published

1520 points

  • Bycasedog21
  • TypeTutorial
  • Posted on2012-02-14 at 05:53:26
The 6120xp switches seem to have a bug when you create a fiber port channel when you have a UCS fabric interconnects talking to them.  If you follow the Cisco guide for the UCS, the FC Port channel will never come up and it will say that there are no members available. So, after a week of beating my head against the wall I came up with a method and so far has worked every time.  I think the main issue is the ports never reset their port state until you specifically disable the ports and then bring them back up again.  So here are my steps as follows.


If the instruction starts with UCS then you need to do the step on the UCS, if it starts with MDS then you need to do it on the fiber channel switches.

1) Install Gbics and get the physical connectivity going.
2) UCS Config the new ports as FC uplink ports (May already be done).
3) Open Cisco Device Manager on log into the appropriate switch.
4) MDS (Device Manager)  click on interface then portchannel.
5) MDS Click Create.
6) MDS increment the ID by one from last portchannel if not done already.
7) MDS set "nontrunk, auto, force, portvsan # depending on the switch, then hit the ... to select interfaces that are in the port channel.
8) UCS create port channel do not add ports then disable it then set the appropriate vsan.
9) UCS add ports.
10) MDS while holding control.. Select all ports in the portchannel then right click one of the selected ports and hit config.
11) MDS in the popup window set the status admin on the ports to down then hit apply.
12) UCS enable port channel.
13) MDS set status admin to up on the ports in the popup window and hit apply. In a about 2 minutes the port channel should come up.

Open your favorite SSH client and connect to the MDS switch that you are working with.  Log in with your username and password. Then type "term mon" to see the connection state while you are doing these steps. It will show in real time as you bring the ports up and down and show you the errors that are related to why the ports are working.  The console will refresh faster than the device manager GUI.

So once you have gone through these steps you can see where the ports have come up. Once you go through these steps you can take the port channel up and down as needed. It seems that the initial build out of the port channel is the issue and the ports on the MDS never go completely down and come back up without doing these steps.
Asked On
2012-02-14 at 05:53:26ID9633
Tags

UCS

,

6120xp

,

fiber

,

portchannel

,

port channel

,

fiberchannel

Topic

Computer Servers

Views
850

Comments

Expert Comment

by: NAMEWITHELD12 on 2012-02-15 at 07:26:47ID: 43035

great article !

I am currently attempting to connect a cisco fabic interconnect to a MDS switch and then to an emc clarion CX 4480 via FC only .

to do this I am thinking that the first step is to set up the interconnect and then in your  step to is to configure the interconnect FC ports as FC uplink ports ,

I am very new to this and could use any advice you can give!

Author Comment

by: casedog21 on 2012-02-15 at 09:08:26ID: 43036

Are you using multiple FI's and MDS's for Redundancy?  What model MDS I heard other models do not have this problem.  

Keep in mind (from what I have read and understand, feel free to correct me) that the failover for the FC is done via the server only.  In my example what I did is created a FC portchannel on each FI.  I have 2 vSan's 100 and 200. My MDS switch A carries all the 100 traffic and B carries all the 200 traffic.  So when I configure my UCS Portchannels FI A is set to vSAN 100 and portchannel on FI B is set to vSAN 200 and connected to their respective MDS switch.

So my 2 Gbics are 4 gig a piece in the portchannel. My sans have 2 adapters both 4 gig. Both sans are connected to both MDS switches for redundant paths.

Word of advice.. Build a server template with all vNic's vHBA's and everything you need. It will save some of the confusion later.

Create your MAC pools for your vNics and vHBA's.  So you can call one HBA_FabA and HBA_FABB. When it asks your for the naming scheme change one of the digits to A on the A side and B on the B side.  It makes it a lot easier to make sure your connections are connecting on the right vSan's / Adapters and vNics and what not.  

If I go into vmware I can look at the MAC for the HBA and can tell what vSAN it is on or if it was mapped incorrectly in the Service Profile that was deployed for that server.

The Ethernet port channel in version 1.43 of the UCS manager had issues as well. The only way I got that to work was to create it command line. Once I update to 2.01(t) all was right with the world. Make sure you set the vNics to their respective FI and enable failover. Did a failover test with a VM running on VMware and it worked like a champ. I was pinging it and only lost one ping. Also had a short hiccup on the server I was logged into while it was failing over. Both for the Uplink Ethernet Portchannel and the FC portchannel.

A tip for performance I have found in vmware is to manage the Datastore Paths and set them to Round Robin. Else it will only use one path and use the secondary as a failover only. I didnt want that. I wanted all links to be utilized equally. The UCS/FI's/MDS's all seemed to handle that well. I migrated (vStorageMotion) 33gig of data from one SAN to the other in under 3 minutes. I was quite pleased.

Getting started on the UCS is quite overwhelming at first. Eventually it all makes sense though.  Not to mention that nothing is really real except the actual physical connections. Even the memory is virtualized to the CPU.

I am still learning everything myself but I think I have it almost completely figured out.

Storage.pdf
  • 54 KB
  • UCS layout
Storage.pdf


See the above for my diagram.

Good luck!

    Add your Comment

    Please Sign up or Log in to comment on this article.

    Join Experts Exchange Today

    Gain Access to all our Tech Resources

    Get personalized answers

    Ask unlimited questions

    Access Proven Solutions

    Search 3.2 million solutions

    Read In-Depth How-To Guides

    1000+ articles, demos, & tips

    Watch Step by Step Tutorials

    Learn direct from top tech pros

    And Much More!

    Your complete tech resource

    See Plans and Pricing

    30-day free trial. Register in 60 seconds.

    Loading Advertisement...

    Top Server Hardware Experts

    1. andyalder

      343,946

      Wizard

      6,464 points yesterday

      Profile
      Rank: Genius
    2. hanccocka

      169,853

      Guru

      1,800 points yesterday

      Profile
      Rank: Genius
    3. dlethe

      134,276

      Master

      750 points yesterday

      Profile
      Rank: Genius
    4. Callandor

      66,040

      Master

      0 points yesterday

      Profile
      Rank: Genius
    5. PowerEdgeTech

      51,546

      Master

      0 points yesterday

      Profile
      Rank: Genius
    6. charlestasse

      49,038

      0 points yesterday

      Profile
      Rank: Wizard
    7. arnold

      37,415

      0 points yesterday

      Profile
      Rank: Genius
    8. kevinhsieh

      34,882

      0 points yesterday

      Profile
      Rank: Genius
    9. noxcho

      31,714

      668 points yesterday

      Profile
      Rank: Genius
    10. paulsolov

      26,304

      0 points yesterday

      Profile
      Rank: Genius
    11. leew

      22,982

      560 points yesterday

      Profile
      Rank: Savant
    12. rindi

      22,689

      0 points yesterday

      Profile
      Rank: Savant
    13. BigSchmuh

      22,258

      0 points yesterday

      Profile
      Rank: Sage
    14. meyersd

      20,732

      0 points yesterday

      Profile
      Rank: Genius
    15. demazter

      20,140

      0 points yesterday

      Profile
      Rank: Genius
    16. pgm554

      18,510

      0 points yesterday

      Profile
      Rank: Sage
    17. IT-Monkey-Dave

      18,418

      0 points yesterday

      Profile
      Rank: Wizard
    18. coolsport00

      18,036

      1,400 points yesterday

      Profile
      Rank: Genius
    19. jamietoner

      16,216

      0 points yesterday

      Profile
      Rank: Genius
    20. SelfGovern

      15,028

      0 points yesterday

      Profile
      Rank: Wizard
    21. woolmilkporc

      14,532

      0 points yesterday

      Profile
      Rank: Genius
    22. dariusg

      14,160

      0 points yesterday

      Profile
      Rank: Genius
    23. Syed_M_Usman

      12,008

      0 points yesterday

      Profile
      Rank: Wizard
    24. chakko

      11,940

      0 points yesterday

      Profile
      Rank: Genius
    25. cwstad2

      11,832

      0 points yesterday

      Profile
      Rank: Guru

    Hall Of Fame