Williams225
asked on
Sun Storagetek 2540 failed to start
Dear All,
I need your help to solve my issue.
I have one Storage baie ( Sun Storagetek 2540) connected to an Esx host (Sun Fire 4150).
The storage lost connection with the CAM (common array management) and the host.
As I cannot connect remotetly to the storage I tried a serial connection to check what's wrong.
But once connected to the storage serial port, what I can see is the bellow logs which appear continuously and cannot have prompt to check anything
03/25/15-13:32:03 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:05 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:07 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:09 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:11 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:13 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:15 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:17 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:19 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:21 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
I restarted the storage but the same logs in attachement after reboot.
Attaching interface lo0... done
Adding 9765 symbols for standalone.
Error
03/25/15-13:51:57 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00
Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
5900 Ethernet 91c111 #1
01 Register read Passed
02 Register test Passed
3A00 NAND Flash
06 Bad Blocks Test Passed
2310 Application Accelerator Unit
01 AAU Register Test Passed
6D00 LSI SAS 1068 IOC--Base Board
01 IOC Register Read Test Passed
02 IOC Register Address Lines Test Passed
03 IOC Register Data Lines Test Passed
65B1 Host Channel 2--Tachyon DX4 Plus
01 TachLite Register Test Passed
65B2 Host Channel 3--Tachyon DX4 Plus
01 TachLite Register Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
Current date: 03/25/15 time: 13:52:18
Send <BREAK> for Service Interface or baud rate change
03/25/15-13:52:19 (GMT) (tRAID): NOTE: Set Powerup State
03/25/15-13:52:19 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0
03/25/15-13:52:19 (GMT) (tRAID): NOTE: SOD: removed SAS host from index 0
03/25/15-13:52:20 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
03/25/15-13:52:20 (GMT) (tRAID): NOTE: lost persistent dq data because buffer was modified or size changed.
esmc0: LinkUp event
03/25/15-13:52:23 (GMT) (tNetCfgInit): NOTE: Network Ready
03/25/15-13:52:24 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup
03/25/15-13:52:27 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-63-00
03/25/15-13:52:34 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:0 prevNumActivePhys:2 numActivePhys:2
03/25/15-13:52:35 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:1 prevNumActivePhys:2 numActivePhys:2
03/25/15-13:52:44 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled
03/25/15-13:52:45 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete
03/25/15-13:52:45 (GMT) (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels
03/25/15-13:52:45 (GMT) (tRAID): NOTE: LockMgr Role is Master
03/25/15-13:52:45 (GMT) (tRAID): WARN: FBM:validateSubModel: Exception - Alt controller not ready
03/25/15-13:52:45 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawned
03/25/15-13:52:45 (GMT) (tRAID): NOTE: spmEarlyData: No data available
03/25/15-13:52:45 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c
03/25/15-13:52:45 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x500a0b85 Low = xbd707010
03/25/15-13:52:51 (GMT) (tSasDiscCom): WARN: SAS: Initial Discovery Complete Time: 30 seconds
03/25/15-13:52:51 (GMT) (tRAID): NOTE: WWN baseName 000200a0-b85bd647 (valid==>SigMatch)
03/25/15-13:52:51 (GMT) (tRAID): NOTE: IonMgr: Host Interface Enabled
03/25/15-13:52:51 (GMT) (tRAID): NOTE: SOD: Pre-Initialization Phase Complete
03/25/15-13:52:51 (GMT) (tRAID): WARN: BID: initialize(): Power latched!
03/25/15-13:52:51 (GMT) (tRAID): WARN: Battery 0's Age has exceeded specified limit.
03/25/15-13:52:52 (GMT) (utlTimer): NOTE: fcnChannelReport ==> -2 -3
03/25/15-13:52:57 (GMT) (tRAID): NOTE: releasing alt ctl from reset
03/25/15-13:52:57 (GMT) (utlTimer): NOTE: fcnChannelReport ==> =2 =3
03/25/15-13:52:58 (GMT) (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0
03/25/15-13:52:58 (GMT) (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0
03/25/15-13:52:58 (GMT) (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating.
03/25/15-13:52:58 (GMT) (tRAID): NOTE: SOD: Code Synchronization Initialization Phase Complete
03/25/15-13:52:58 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleExceptio n Error in iop::requestAltIopDelay
03/25/15-13:52:58 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSendInfeasibleExcepti on Error)
03/25/15-13:52:58 (GMT) (NvpsPersistentSyncM): NOTE: NVSRAM Persistent Storage updated successfully
03/25/15-13:52:58 (GMT) (tRAID): NOTE: USM Mgr initialization complete with 0 records.
03/25/15-13:52:59 (GMT) (tRAID): WARN: spm: unable to exchange features, assuming none
03/25/15-13:52:59 (GMT) (tRAID): NOTE: SPM acquireObjects exception: IconSendInfeasibleExceptio n Error
03/25/15-13:52:59 (GMT) (tRAID): NOTE: DBRead 0.133 secs
03/25/15-13:52:59 (GMT) (tRAID): NOTE: fcn: Peering Disabled (Alt Unavailable)
03/25/15-13:52:59 (GMT) (tRAID): NOTE: sas: Peering Disabled (Alt Unavailable)
03/25/15-13:52:59 (GMT) (tRAID): NOTE: ion: Peering Disabled (Alt Unavailable)
03/25/15-13:53:00 (GMT) (tRAID): NOTE: PM - reading DB (records 1..0)
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSendInfeasibleExcepti on Error)
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: validateCacheMem() cache memory is invalid
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: validateCacheMem() Initializing my partition
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: sodReclaimRecovery() interrupted reclaim was complete
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: sodReclaimRecovery() releasing alternate
03/25/15-13:53:00 (GMT) (tRAID): NOTE: releasing alt ctl from reset
03/25/15-13:53:01 (GMT) (tRAID): WARN: CCM: mirrorInit() alternate did not check in
03/25/15-13:53:01 (GMT) (tRAID): NOTE: CCM: initialize(): Configuring cache
03/25/15-13:53:01 (GMT) (tRAID): WARN: Failed to send a powerup message packet - status: -1
03/25/15-13:53:01 (GMT) (tRAID): NOTE: Starting UWManager::initialize, entries 510, invalid index -1
03/25/15-13:53:01 (GMT) (tRAID): NOTE: Size of NVSRAM IW Queue is 0
03/25/15-13:53:02 (GMT) (tRAID): NOTE: CCM: initComplete() interrupted reclaim complete
03/25/15-13:53:02 (GMT) (tRAID): NOTE: CCM: initComplete() releasing alternate
03/25/15-13:53:02 (GMT) (tRAID): NOTE: releasing alt ctl from reset
03/25/15-13:53:03 (GMT) (tRAID): NOTE: RTR: IO Released
03/25/15-13:53:03 (GMT) (tRAID): NOTE: DiagVolManager::initialize : Exception - Alt controller not ready
03/25/15-13:53:03 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleExceptio n Error in iop::requestAltIopDelay
03/25/15-13:53:03 (GMT) (tRAID): NOTE: SOD: Initialization Phase Complete
========================== ========== ==========
Title: Disk Array Controller
Copyright 2005-2009 LSI Logic Corporation, All Rights Reserved.
Name: RC
Version: 07.35.44.10
Date: 04/07/2009
Time: 22:45:17 CDT
Models: 1932
Manager: devmgr.v1035api01.Manager
========================== ========== ==========
03/25/15-13:32:03 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:05 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:07 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:09 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:11 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:13 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:15 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:17 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:19 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
03/25/15-13:32:21 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand lingFeasib le caught IconSendInfeasibleExceptio n Error
I need your idea about the root cause and how can i resolved it.
Best regards.
logs-storagetek-2540.txt
I need your help to solve my issue.
I have one Storage baie ( Sun Storagetek 2540) connected to an Esx host (Sun Fire 4150).
The storage lost connection with the CAM (common array management) and the host.
As I cannot connect remotetly to the storage I tried a serial connection to check what's wrong.
But once connected to the storage serial port, what I can see is the bellow logs which appear continuously and cannot have prompt to check anything
03/25/15-13:32:03 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:05 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:07 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:09 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:11 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:13 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:15 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:17 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:19 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:21 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
I restarted the storage but the same logs in attachement after reboot.
Attaching interface lo0... done
Adding 9765 symbols for standalone.
Error
03/25/15-13:51:57 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00
Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
5900 Ethernet 91c111 #1
01 Register read Passed
02 Register test Passed
3A00 NAND Flash
06 Bad Blocks Test Passed
2310 Application Accelerator Unit
01 AAU Register Test Passed
6D00 LSI SAS 1068 IOC--Base Board
01 IOC Register Read Test Passed
02 IOC Register Address Lines Test Passed
03 IOC Register Data Lines Test Passed
65B1 Host Channel 2--Tachyon DX4 Plus
01 TachLite Register Test Passed
65B2 Host Channel 3--Tachyon DX4 Plus
01 TachLite Register Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
Current date: 03/25/15 time: 13:52:18
Send <BREAK> for Service Interface or baud rate change
03/25/15-13:52:19 (GMT) (tRAID): NOTE: Set Powerup State
03/25/15-13:52:19 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0
03/25/15-13:52:19 (GMT) (tRAID): NOTE: SOD: removed SAS host from index 0
03/25/15-13:52:20 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
03/25/15-13:52:20 (GMT) (tRAID): NOTE: lost persistent dq data because buffer was modified or size changed.
esmc0: LinkUp event
03/25/15-13:52:23 (GMT) (tNetCfgInit): NOTE: Network Ready
03/25/15-13:52:24 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup
03/25/15-13:52:27 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-63-00
03/25/15-13:52:34 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:0 prevNumActivePhys:2 numActivePhys:2
03/25/15-13:52:35 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:1 prevNumActivePhys:2 numActivePhys:2
03/25/15-13:52:44 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled
03/25/15-13:52:45 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete
03/25/15-13:52:45 (GMT) (tRAID): WARN: No attempt made to open Inter-Controller Communication Channels
03/25/15-13:52:45 (GMT) (tRAID): NOTE: LockMgr Role is Master
03/25/15-13:52:45 (GMT) (tRAID): WARN: FBM:validateSubModel: Exception - Alt controller not ready
03/25/15-13:52:45 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawned
03/25/15-13:52:45 (GMT) (tRAID): NOTE: spmEarlyData: No data available
03/25/15-13:52:45 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c
03/25/15-13:52:45 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x500a0b85 Low = xbd707010
03/25/15-13:52:51 (GMT) (tSasDiscCom): WARN: SAS: Initial Discovery Complete Time: 30 seconds
03/25/15-13:52:51 (GMT) (tRAID): NOTE: WWN baseName 000200a0-b85bd647 (valid==>SigMatch)
03/25/15-13:52:51 (GMT) (tRAID): NOTE: IonMgr: Host Interface Enabled
03/25/15-13:52:51 (GMT) (tRAID): NOTE: SOD: Pre-Initialization Phase Complete
03/25/15-13:52:51 (GMT) (tRAID): WARN: BID: initialize(): Power latched!
03/25/15-13:52:51 (GMT) (tRAID): WARN: Battery 0's Age has exceeded specified limit.
03/25/15-13:52:52 (GMT) (utlTimer): NOTE: fcnChannelReport ==> -2 -3
03/25/15-13:52:57 (GMT) (tRAID): NOTE: releasing alt ctl from reset
03/25/15-13:52:57 (GMT) (utlTimer): NOTE: fcnChannelReport ==> =2 =3
03/25/15-13:52:58 (GMT) (tRAID): NOTE: ACS: Icon ping to alternate failed: -2, resp: 0
03/25/15-13:52:58 (GMT) (tRAID): NOTE: ACS: autoCodeSync(): Process start. Comm Mode: 0, Status: 0
03/25/15-13:52:58 (GMT) (tRAID): WARN: ACS: autoCodeSync(): Skipped since alt not communicating.
03/25/15-13:52:58 (GMT) (tRAID): NOTE: SOD: Code Synchronization Initialization Phase Complete
03/25/15-13:52:58 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleExceptio
03/25/15-13:52:58 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSendInfeasibleExcepti
03/25/15-13:52:58 (GMT) (NvpsPersistentSyncM): NOTE: NVSRAM Persistent Storage updated successfully
03/25/15-13:52:58 (GMT) (tRAID): NOTE: USM Mgr initialization complete with 0 records.
03/25/15-13:52:59 (GMT) (tRAID): WARN: spm: unable to exchange features, assuming none
03/25/15-13:52:59 (GMT) (tRAID): NOTE: SPM acquireObjects exception: IconSendInfeasibleExceptio
03/25/15-13:52:59 (GMT) (tRAID): NOTE: DBRead 0.133 secs
03/25/15-13:52:59 (GMT) (tRAID): NOTE: fcn: Peering Disabled (Alt Unavailable)
03/25/15-13:52:59 (GMT) (tRAID): NOTE: sas: Peering Disabled (Alt Unavailable)
03/25/15-13:52:59 (GMT) (tRAID): NOTE: ion: Peering Disabled (Alt Unavailable)
03/25/15-13:53:00 (GMT) (tRAID): NOTE: PM - reading DB (records 1..0)
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CheckInMonitor: Check-in failed (IconSendInfeasibleExcepti
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: validateCacheMem() cache memory is invalid
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: validateCacheMem() Initializing my partition
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: sodReclaimRecovery() interrupted reclaim was complete
03/25/15-13:53:00 (GMT) (tRAID): NOTE: CCM: sodReclaimRecovery() releasing alternate
03/25/15-13:53:00 (GMT) (tRAID): NOTE: releasing alt ctl from reset
03/25/15-13:53:01 (GMT) (tRAID): WARN: CCM: mirrorInit() alternate did not check in
03/25/15-13:53:01 (GMT) (tRAID): NOTE: CCM: initialize(): Configuring cache
03/25/15-13:53:01 (GMT) (tRAID): WARN: Failed to send a powerup message packet - status: -1
03/25/15-13:53:01 (GMT) (tRAID): NOTE: Starting UWManager::initialize, entries 510, invalid index -1
03/25/15-13:53:01 (GMT) (tRAID): NOTE: Size of NVSRAM IW Queue is 0
03/25/15-13:53:02 (GMT) (tRAID): NOTE: CCM: initComplete() interrupted reclaim complete
03/25/15-13:53:02 (GMT) (tRAID): NOTE: CCM: initComplete() releasing alternate
03/25/15-13:53:02 (GMT) (tRAID): NOTE: releasing alt ctl from reset
03/25/15-13:53:03 (GMT) (tRAID): NOTE: RTR: IO Released
03/25/15-13:53:03 (GMT) (tRAID): NOTE: DiagVolManager::initialize
03/25/15-13:53:03 (GMT) (tRAID): NOTE: Caught IconSendInfeasibleExceptio
03/25/15-13:53:03 (GMT) (tRAID): NOTE: SOD: Initialization Phase Complete
==========================
Title: Disk Array Controller
Copyright 2005-2009 LSI Logic Corporation, All Rights Reserved.
Name: RC
Version: 07.35.44.10
Date: 04/07/2009
Time: 22:45:17 CDT
Models: 1932
Manager: devmgr.v1035api01.Manager
==========================
03/25/15-13:32:03 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:05 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:07 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:09 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:11 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:13 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:15 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:17 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:19 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
03/25/15-13:32:21 (GMT) (ccmEventTask): NOTE: CacheReconfigEvent::isHand
I need your idea about the root cause and how can i resolved it.
Best regards.
logs-storagetek-2540.txt
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thank you for your feedback.
I will try to replace the controller and check again.
Regards