We help IT Professionals succeed at work.

HP San Expert Needed! MSA Firmware Upgrade Causing Issues

epaschal
epaschal asked
on
Hello Experts,

I have inherited a SAN configuration and am in need of some expert assistance.

I have an HP MSA 2312i SAN that consist of two enclosures which provide storage for my XenServers.

One of the drives in the second enclosure was generating errors, so HP had me install O320B02 in the Storage Management Utility website under Tools > Update Firmware > Install Expansion Module Firmware.

Now that table reads:
Enclosure ID Enclosure Name Module ID Module Position Module Part Number Module Code Version
 2 Enclosure 2 A Top AJ751A 3202
 2 Enclosure 2 B Bottom AJ751A 3202

During the firmware update the system became unavailable, however HP had me power cycle and it appears that the new firmware (3202) is showing on that table.

However after doing the update, my XenServers are acting crazy and I'm seeing all sorts of issues. Plus oddly one of the power supplies is faulting now, for which they are shipping a replacement.

I noticed that when I view the enclosures, the EMP information no longer matches (it previously did) and I wonder is this is an issue.

************
Enclosure ID 1
************
Vendor HP
Model SPS Chassis
Disk Slots 12
Part Number 481321-001
Manufacturing Date 2009-10-21 16:16:17
Manufacturing Location Shunde,CN
Revision A
EMP A Revision 1112    <<<<<<<<
EMP B Revision 1112     <<<<<<<<
EMP A Bus ID 0
EMP B Bus ID 1
EMP A Target ID 31
EMP B Target ID 31

**************
Enclosure ID 2
**************
Vendor HP
Model SPS Chassis
Disk Slots 12
Part Number 481321-001
Manufacturing Date 2009-08-15 00:23:07
Manufacturing Location Shunde,CN
Revision A
EMP A Revision 3202   <<<<<<<<
EMP B Revision 3202    <<<<<<<<
EMP A Bus ID 0
EMP B Bus ID 1
EMP A Target ID 63
EMP B Target ID 63


Previously both of these matched at 1112, now one says 3202 and the other 1112.

HP is saying that the first enclosure firmware hasn't upgraded, however I don't see an option for doing this. Is this an issue that they don't match, and if so, how do I remedy?

I'm concerned because before this upgrade all of my xenservers were working just fine, and how it's chaos.


In case you need it - here's the info on the controller modules:

Current Versions of Controller Module A
Property Value
Storage Controller Code Version M112R14
Memory Controller FPGA Code Version F300R22
Storage Controller Loader Code Version 19.009
Management Controller Code Version W441R39
Management Controller Loader Code Version 12.015
Expander Controller Code Version 1112
CPLD Code Version 8
 
Current Versions of Controller Module B
Property Value
Storage Controller Code Version M112R14
Memory Controller FPGA Code Version F300R22
Storage Controller Loader Code Version 19.009
Management Controller Code Version W441R39
Management Controller Loader Code Version 12.015
Expander Controller Code Version 1112
CPLD Code Version 8
 


Thanks!


Comment
Watch Question

I am not an MSA expert but i do work with P4000 and other SAN products.  I would not recommend SAN nodes with different firmware on them.  Did you shut down the XenServers when the SAN's were being upgraded?  Are you running HA on the XenServers?  Have you rebooted everything at least once?

Author

Commented:
Multiple reboots. All servers were shut down prior to the upgrade. I normally have HA on but had disabled it for the shutdown and tried with and without with no noticeable improvement.
Distinguished Expert 2019
Commented:
>HP is saying that the first enclosure firmware hasn't upgraded, however I don't see an option for doing this.

The update for the main controllers is meant to update any enclosures EMP after doing the main unit, have you tried to repeat it? Although HP asked you to upgrade Install Expansion Module Firmware you could have done upgrade controller firmware, that way the controller updates (10 minutes) and then the EMPs (3 mins each). I must admit I haven't had fun updating MSA2000 firmware either, sometimes given up with doing it with the web GUI and run the executable on a PC that pushes it out instead and left the PC running overnight - thought it had hung.

Can't find the G2 SMU reference guide, the G3 might help at http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02520791/c02520791.pdf
Commented:
Hi, I used to be an HP SAN expert, but I left HP some time ago, but my good friend is currently the goto guy for HP in this area, so I asked him for you.

He said that he has seen this issue before, and what he recommends is the following:

1.) Make sure all servers are down.
2.) Shut down the MSA
3.) Pull the controller that was successfully updated.
4.) Bring up the MSA with just the controller that failed to update
5.) Update the controller.

Assuming the Update takes which he thinks it will with just the one controller, re down the MSA, reinstall the other controller, bring up the MSA and all should be well with the world.

If it still doesn't take, he says there is a hidden reset switch on the back of the controller which you can try pushing.  He cautioned that this is not a "use it whenever" type switch, but this is a last ditch before the controller needs to be replaced, so if it still doesn't take, you can reset it, and try again.

Hope this helps!

Author

Commented:
Thanks for the advice. To help illustrate what's going on, I've attached the following.  firmware.pdf
I'm going the same route as everyone on this, having to deal with MSAs, EVAs and now P6000 , powerdown everything, and redo the update... this is how I usually do when controllers seems to be "in need of attention". Experience also tells me if you can, reboot the damn thing before and update after... this seems to somehow help getting rid of firmware updates that fails or hang mysteriously. And no need to mention that backups of everything is more than crucial when doing updates ;). As always provided as is no warranty as even trained professionals can "brick" systems with a faulty update :P
Commented:
HP is stating that this is a supported configuration even though it's never been like this before. Therefore I just have to leave it alone.

Author

Commented:
No clear resolution at this time.