Solved

Failure in controller of HP StorageWorks MSA2012fc

Posted on 2014-10-15
8
821 Views
Last Modified: 2014-10-17
Hi experts, I have the next critical error on MSA logs (attached):
TUE OCT 07 11:40:55 2014 [314] #A16253: MSA2012fc Array SN#00C0FFD839E1 Controller A CRITICAL: FRU type: RAID IOM B, problem: encl 0. Product ID: AJ744A, S/N: 3CL922S426 rev: H.  Related event ID: 10016252, type: 313

Open in new window

The array connects to an Oracle RAC server with two instances
The AlertLog of Oracle Instance #2 says:
Errors in file /cots/oracle/app/oracle/admin/xa21/bdump/xa212_j001_30215.trc:
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1529712
Additional information: -1

Open in new window

The Oracle instance #1 does not report errors.
The version of S.O is:
uname -a
Linux 2.6.18-53.1.21.el5 #1 SMP Wed May 7 08:42:34 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

Open in new window

and the versions in MSA:
# versions
Controller A Versions
---------------------
Storage Controller CPU Type   : Celeron 566MHz
Storage Controller Firmware   : J200P30
Storage Controller Memory     : F300R22
Storage Controller Loader     : 15.010
Management Controller Firmware: W420R52
Management Controller Loader  : 12.013
Expander Controller Firmware  : 3022
CPLD Revision                 : 27
Hardware Revision             : LCA 56
Host Interface Module         : 26
Host Interface Module Model   : 1

Controller B Versions
---------------------
Storage Controller CPU Type   : Celeron 566MHz
Storage Controller Firmware   : J200P30
Storage Controller Memory     : F300R22
Storage Controller Loader     : 15.010
Management Controller Firmware: W420R52
Management Controller Loader  : 12.013
Expander Controller Firmware  : 3022
CPLD Revision                 : 27
Hardware Revision             : LCA 56
Host Interface Module         : 26
Host Interface Module Model   : 1

Open in new window

Could you please suggest me the steps to find the problem and its resolution?
Thankyou
Regards
logs-from-20140804.log
0
Comment
Question by:carlino70
  • 4
  • 3
8 Comments
 
LVL 42

Assisted Solution

by:paulsolov
paulsolov earned 250 total points
ID: 40383604
I would give HP a call and send over the logs.  Take a look at your fibre channel paths on the servers to see if all paths are showing, most likely the controller has failed over but you should look at getting a controller replaced if it's a hardware issue
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40383856
When you look at your MSA, do you have one or two controllers ?
Don't you see any orange light / LED message on it ?
0
 

Author Comment

by:carlino70
ID: 40384207
paulsolov, the warranty expired. I searched for similar cases in the HP forums with various attempted solutions, with varying results.
0
 

Author Comment

by:carlino70
ID: 40384224
Stampel, the MSA has 2 controllers.
No colors orange light / display in the Led's
Currently the leds 'LINK' of each input FO are off, on controller #1. But I see information from A y B, with up/down link, permanently:
TUE OCT 14 14:28:09 2014 [111] #A16361: MSA2012fc Array SN#00C0FFD839E1 Controller A INFORMATIONAL: Host link up Chan1: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #A16360: MSA2012fc Array SN#00C0FFD839E1 Controller A WARNING: Host link down Chan1
TUE OCT 14 14:28:08 2014 [111] #A16359: MSA2012fc Array SN#00C0FFD839E1 Controller A INFORMATIONAL: Host link up Chan0: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #A16358: MSA2012fc Array SN#00C0FFD839E1 Controller A WARNING: Host link down Chan0
TUE OCT 14 14:28:08 2014 [111] #B10072: MSA2012fc Array SN#00C0FFD839E1 Controller B INFORMATIONAL: Host link up Chan0: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #B10071: MSA2012fc Array SN#00C0FFD839E1 Controller B WARNING: Host link down Chan0
TUE OCT 14 14:28:08 2014 [111] #B10070: MSA2012fc Array SN#00C0FFD839E1 Controller B INFORMATIONAL: Host link up Chan1: 4 Loop IDs, External Device(s)
TUE OCT 14 14:28:08 2014 [112] #B10069: MSA2012fc Array SN#00C0FFD839E1 Controller B WARNING: Host link down Chan1

Open in new window

0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 
LVL 7

Accepted Solution

by:
Stampel earned 250 total points
ID: 40384464
Did/could you upgrade your controllers to latest firmware first ?
0
 

Author Comment

by:carlino70
ID: 40384495
not yet. I wanted to get an accurate diagnosis before
0
 
LVL 7

Expert Comment

by:Stampel
ID: 40386924
The real important message i can see is from your logs is :
"CRITICAL: RAID controller B failed, reason PCIE link recovery failed" + Failover ...
MSA2012fc and MSA2000 have had many problems, i would upgrade firmware first just in case.

Do you still have File I/O error on Oracle RAC ?
0
 

Author Comment

by:carlino70
ID: 40387239
Stampel, I/O Errors no longer appear.
But the controller is working like simple configuration.
I will be doing a firmware upgrade, and evenual replacement controller.
Thanks for your comments.
Regards.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Bellevue, WA, USA; Late December, 1980 "OK, folks, we nailed the IBM contract. We're going to have many meetings like this to discuss features and functions...and how to move from 86-DOS, the CP/M clone we just purchased, to our baby, MS-DOS. Thi…
The Samsung SSD 840 EVO and 840 EVO mSATA have a well-known problem with a drop in read performance. I first learned about this in an interesting thread here at Experts Exchange: http://www.experts-exchange.com/Hardware/Storage/Hard_Drives/Q_2852…
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now