Solved

Failure in controller of HP StorageWorks MSA2012fc

Posted on 2014-10-15
8
779 Views
Last Modified: 2014-10-17
Hi experts, I have the next critical error on MSA logs (attached):
TUE OCT 07 11:40:55 2014 [314] #A16253: MSA2012fc Array SN#00C0FFD839E1 Controller A CRITICAL: FRU type: RAID IOM B, problem: encl 0. Product ID: AJ744A, S/N: 3CL922S426 rev: H.  Related event ID: 10016252, type: 313

Open in new window

The array connects to an Oracle RAC server with two instances
The AlertLog of Oracle Instance #2 says:
Errors in file /cots/oracle/app/oracle/admin/xa21/bdump/xa212_j001_30215.trc:
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1529712
Additional information: -1

Open in new window

The Oracle instance #1 does not report errors.
The version of S.O is:
uname -a
Linux 2.6.18-53.1.21.el5 #1 SMP Wed May 7 08:42:34 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

Open in new window

and the versions in MSA:
# versions
Controller A Versions
---------------------
Storage Controller CPU Type   : Celeron 566MHz
Storage Controller Firmware   : J200P30
Storage Controller Memory     : F300R22
Storage Controller Loader     : 15.010
Management Controller Firmware: W420R52
Management Controller Loader  : 12.013
Expander Controller Firmware  : 3022
CPLD Revision                 : 27
Hardware Revision             : LCA 56
Host Interface Module         : 26
Host Interface Module Model   : 1

Controller B Versions
---------------------
Storage Controller CPU Type   : Celeron 566MHz
Storage Controller Firmware   : J200P30
Storage Controller Memory     : F300R22
Storage Controller Loader     : 15.010
Management Controller Firmware: W420R52
Management Controller Loader  : 12.013
Expander Controller Firmware  : 3022
CPLD Revision                 : 27
Hardware Revision             : LCA 56
Host Interface Module         : 26
Host Interface Module Model   : 1

Open in new window

Could you please suggest me the steps to find the problem and its resolution?
Thankyou
Regards
logs-from-20140804.log
0
Comment
Question by:carlino70
  • 4
  • 3
8 Comments
 
LVL 42

Assisted Solution

by:paulsolov
paulsolov earned 250 total points
Comment Utility
I would give HP a call and send over the logs.  Take a look at your fibre channel paths on the servers to see if all paths are showing, most likely the controller has failed over but you should look at getting a controller replaced if it's a hardware issue
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
When you look at your MSA, do you have one or two controllers ?
Don't you see any orange light / LED message on it ?
0
 

Author Comment

by:carlino70
Comment Utility
paulsolov, the warranty expired. I searched for similar cases in the HP forums with various attempted solutions, with varying results.
0
 

Author Comment

by:carlino70
Comment Utility
Stampel, the MSA has 2 controllers.
No colors orange light / display in the Led's
Currently the leds 'LINK' of each input FO are off, on controller #1. But I see information from A y B, with up/down link, permanently:
TUE OCT 14 14:28:09 2014 [111] #A16361: MSA2012fc Array SN#00C0FFD839E1 Controller A INFORMATIONAL: Host link up Chan1: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #A16360: MSA2012fc Array SN#00C0FFD839E1 Controller A WARNING: Host link down Chan1
TUE OCT 14 14:28:08 2014 [111] #A16359: MSA2012fc Array SN#00C0FFD839E1 Controller A INFORMATIONAL: Host link up Chan0: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #A16358: MSA2012fc Array SN#00C0FFD839E1 Controller A WARNING: Host link down Chan0
TUE OCT 14 14:28:08 2014 [111] #B10072: MSA2012fc Array SN#00C0FFD839E1 Controller B INFORMATIONAL: Host link up Chan0: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #B10071: MSA2012fc Array SN#00C0FFD839E1 Controller B WARNING: Host link down Chan0
TUE OCT 14 14:28:08 2014 [111] #B10070: MSA2012fc Array SN#00C0FFD839E1 Controller B INFORMATIONAL: Host link up Chan1: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #B10069: MSA2012fc Array SN#00C0FFD839E1 Controller B WARNING: Host link down Chan1

Open in new window

0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 7

Accepted Solution

by:
Stampel earned 250 total points
Comment Utility
Did/could you upgrade your controllers to latest firmware first ?
0
 

Author Comment

by:carlino70
Comment Utility
not yet. I wanted to get an accurate diagnosis before
0
 
LVL 7

Expert Comment

by:Stampel
Comment Utility
The real important message i can see is from your logs is :
"CRITICAL: RAID controller B failed, reason PCIE link recovery failed" + Failover ...
MSA2012fc and MSA2000 have had many problems, i would upgrade firmware first just in case.

Do you still have File I/O error on Oracle RAC ?
0
 

Author Comment

by:carlino70
Comment Utility
Stampel, I/O Errors no longer appear.
But the controller is working like simple configuration.
I will be doing a firmware upgrade, and evenual replacement controller.
Thanks for your comments.
Regards.
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

AWS Glacier is Amazons cheapest storage option and is their answer to a ‘Cold’ storage service.  Customers primarily use this service for archival purposes and storage of infrastructure backups.  Its unlimited storage potential and low storage cost …
This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now