asked on

How do i fix cissesrv errors incorrectly reporting disk drive removal

We are using an HP ProLiant ML350G6 server with an embedded P410i smart array controller with 6x146GB disks. Every so often it reports that a disk has been removed and subsequently reinserted, and that it has to rebuild the physical and logical volumes, however the disk has never been removed. Has anyone come across this before and found a resolution?

The full sequence of events is:

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24582
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
A SAS physical drive located in bay 5 was removed. The drive can be found in box 1 which is attached to port 2I of array controller P410i [Embedded].
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24582</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15408</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data> SAS </Data>
<Data>5</Data>
<Data>removed</Data>
<Data>1</Data>
<Data>2I</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24595
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
A drive failure notification has been received for the SAS physical drive located in bay 5. This drive can be found in box 1 which is connected to port 2I of the array controller P410i [Embedded]. The failure reason received from the HP Smart Array firmware is: REMOVED_IN_HOT_PLUG.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="50229">24595</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15409</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data> SAS </Data>
<Data>5</Data>
<Data>1</Data>
<Data>2I</Data>
<Data>REMOVED_IN_HOT_PLUG</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 1 of array controller P410i [Embedded] has encountered a status change from:

Status: OK
to
Status: INTERIM RECOVERY MODE
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15410</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>1</Data>
<Data>OK</Data>
<Data>INTERIM RECOVERY MODE</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 2 of array controller P410i [Embedded] has encountered a status change from:

Status: OK
to
Status: INTERIM RECOVERY MODE
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15411</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>2</Data>
<Data>OK</Data>
<Data>INTERIM RECOVERY MODE</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24582
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
A SAS physical drive located in bay 5 was inserted. The drive can be found in box 1 which is attached to port 2I of array controller P410i [Embedded].
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24582</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15412</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data> SAS </Data>
<Data>5</Data>
<Data>inserted</Data>
<Data>1</Data>
<Data>2I</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 1 of array controller P410i [Embedded] has encountered a status change from:

Status: INTERIM RECOVERY MODE
to
Status: READY FOR RECOVERY
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15413</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>1</Data>
<Data>INTERIM RECOVERY MODE</Data>
<Data>READY FOR RECOVERY</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 2 of array controller P410i [Embedded] has encountered a status change from:

Status: INTERIM RECOVERY MODE
to
Status: READY FOR RECOVERY
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15414</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>2</Data>
<Data>INTERIM RECOVERY MODE</Data>
<Data>READY FOR RECOVERY</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:22:47
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 1 of array controller P410i [Embedded] has encountered a status change from:

Status: READY FOR RECOVERY
to
Status: RECOVERING
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:22:47.000Z" />
<EventRecordID>15415</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>1</Data>
<Data>READY FOR RECOVERY</Data>
<Data>RECOVERING</Data>
</EventData>
</Event>

Log Name: System
Source: HP Smart Array
Date: 08/09/2010 18:23:46
Event ID: 203
Task Category: Primary Storage Device
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
A physical drive status has changed to Rebuilding. Data is now being rebuilt onto this drive from the RAID set.

User Action
No action is recommended.

WBEM Indication Properties
AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_DiskDrive.CreationClassName="HPSA_DiskDrive",DeviceID="5001438007CBDEF0:4",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A physical drive status has changed to Rebuilding. Data is now being rebuilt onto this drive from the RAID set."
DeviceControllerLogicalLocation[0]: "Smart Array P410i in Slot 0"
DeviceModel: "EG0146FAWHU"
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "203"
EventTime: "20100908182338.565000-000"
HWFirmwareVersion[0]: "HPDD"
HWLogicalLocation[0]: "Port:2I Box:1 Bay:5"
HWManufacturer: "HP"
HWPhysicalLocation: "Box 1 Bay 5"
HWSerialNumber[0]: "3SD35GK300009037KLQ5"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{ADC690B6-052B-45FE-A73F-C2A352242973}"
IndicationTime: "20100908182343.573000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 2 0x2 (Information)
ProbableCause: 1 0x1 (Other)
ProbableCauseDescription: "Physical Drive Status Changed to Rebuilding"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "No action is recommended."
Summary: "Physical drive is now rebuilding."
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284402266776594 0x1cb4f7a981c2412
VariableNames[0]: "FailureDescription"
VariableNames[1]: "LastFailureDescription"
VariableNames[2]: "DriveInterface"
VariableTypes[0]: 1 0x1 (string)
VariableTypes[1]: 1 0x1 (string)
VariableTypes[2]: 4 0x4 (uint16)
VariableValues[0]: "Rebuilding"
VariableValues[1]: "Hot removed"
VariableValues[2]: "3"

For more information, please contact HP Support.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="HP Smart Array" />
<EventID Qualifiers="16384">203</EventID>
<Level>4</Level>
<Task>6</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:23:46.000Z" />
<EventRecordID>15416</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>A physical drive status has changed to Rebuilding. Data is now being rebuilt onto this drive from the RAID set.</Data>
<Data>No action is recommended.</Data>
<Data>AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_DiskDrive.CreationClassName="HPSA_DiskDrive",DeviceID="5001438007CBDEF0:4",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A physical drive status has changed to Rebuilding. Data is now being rebuilt onto this drive from the RAID set."
DeviceControllerLogicalLocation[0]: "Smart Array P410i in Slot 0"
DeviceModel: "EG0146FAWHU"
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "203"
EventTime: "20100908182338.565000-000"
HWFirmwareVersion[0]: "HPDD"
HWLogicalLocation[0]: "Port:2I Box:1 Bay:5"
HWManufacturer: "HP"
HWPhysicalLocation: "Box 1 Bay 5"
HWSerialNumber[0]: "3SD35GK300009037KLQ5"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{ADC690B6-052B-45FE-A73F-C2A352242973}"
IndicationTime: "20100908182343.573000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 2 0x2 (Information)
ProbableCause: 1 0x1 (Other)
ProbableCauseDescription: "Physical Drive Status Changed to Rebuilding"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "No action is recommended."
Summary: "Physical drive is now rebuilding."
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284402266776594 0x1cb4f7a981c2412
VariableNames[0]: "FailureDescription"
VariableNames[1]: "LastFailureDescription"
VariableNames[2]: "DriveInterface"
VariableTypes[0]: 1 0x1 (string)
VariableTypes[1]: 1 0x1 (string)
VariableTypes[2]: 4 0x4 (uint16)
VariableValues[0]: "Rebuilding"
VariableValues[1]: "Hot removed"
VariableValues[2]: "3"
</Data>
<Data>For more information, please contact HP Support.</Data>
</EventData>
</Event>

Log Name: System
Source: HP Smart Array
Date: 08/09/2010 18:23:46
Event ID: 103
Task Category: Primary Storage Device
Level: Warning
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
A logical drive is in a degraded state. Device is still servicing I/O but redundancy may be degraded or lost.

User Action
Check status of physical drives. Degraded status may be caused by failed or missing physical drives, or a data recovery operation on one or more physical drives.

WBEM Indication Properties
AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_StorageVolume.CreationClassName="HPSA_StorageVolume",DeviceID="5001438007CBDEF0-LD1",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A logical drive is in a degraded state. Device is still servicing I/O but redundancy may be degraded or lost."
DeviceOperationalStatus[0]: 3 0x3 (Degraded)
DeviceOperationalStatus[1]: 32777 0x8009 (Recovering)
DeviceOperationalStatusPrevious[0]: 2 0x2 (OK)
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "103"
EventTime: "20100908182338.565000-000"
HWLogicalLocation[0]: "Logical Drive 2 (RAID 50)"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{006734A8-977B-4802-B051-1AEF7F34C20A}"
IndicationTime: "20100908182346.708000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 3 0x3 (Degraded/Warning)
ProbableCause: 86 0x56 (Disk Failure)
ProbableCauseDescription: "Logical State Changed to Degraded"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "Check status of physical drives. Degraded status may be caused by failed or missing physical drives, or a data recovery operation on one or more physical drives."
Summary: "Logical drive in degraded state"
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284402268024596 0x1cb4f7a982f2f14
VariableNames[0]: "FailureDescription"
VariableTypes[0]: 1 0x1 (string)
VariableValues[0]: "Recovering"

For more information, please contact HP Support.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="HP Smart Array" />
<EventID Qualifiers="32768">103</EventID>
<Level>3</Level>
<Task>6</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:23:46.000Z" />
<EventRecordID>15417</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>A logical drive is in a degraded state. Device is still servicing I/O but redundancy may be degraded or lost.</Data>
<Data>Check status of physical drives. Degraded status may be caused by failed or missing physical drives, or a data recovery operation on one or more physical drives.</Data>
<Data>AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_StorageVolume.CreationClassName="HPSA_StorageVolume",DeviceID="5001438007CBDEF0-LD1",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A logical drive is in a degraded state. Device is still servicing I/O but redundancy may be degraded or lost."
DeviceOperationalStatus[0]: 3 0x3 (Degraded)
DeviceOperationalStatus[1]: 32777 0x8009 (Recovering)
DeviceOperationalStatusPrevious[0]: 2 0x2 (OK)
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "103"
EventTime: "20100908182338.565000-000"
HWLogicalLocation[0]: "Logical Drive 2 (RAID 50)"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{006734A8-977B-4802-B051-1AEF7F34C20A}"
IndicationTime: "20100908182346.708000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 3 0x3 (Degraded/Warning)
ProbableCause: 86 0x56 (Disk Failure)
ProbableCauseDescription: "Logical State Changed to Degraded"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "Check status of physical drives. Degraded status may be caused by failed or missing physical drives, or a data recovery operation on one or more physical drives."
Summary: "Logical drive in degraded state"
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284402268024596 0x1cb4f7a982f2f14
VariableNames[0]: "FailureDescription"
VariableTypes[0]: 1 0x1 (string)
VariableValues[0]: "Recovering"
</Data>
<Data>For more information, please contact HP Support.</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:23:48
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 1 of array controller P410i [Embedded] has encountered a status change from:

Status: RECOVERING
to
Status: OK
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:23:48.000Z" />
<EventRecordID>15418</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>1</Data>
<Data>RECOVERING</Data>
<Data>OK</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:23:48
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 2 of array controller P410i [Embedded] has encountered a status change from:

Status: READY FOR RECOVERY
to
Status: RECOVERING
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:23:48.000Z" />
<EventRecordID>15419</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>2</Data>
<Data>READY FOR RECOVERY</Data>
<Data>RECOVERING</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 18:53:19
Event ID: 24598
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
Logical drive 2 of array controller P410i [Embedded] has encountered a status change from:

Status: RECOVERING
to
Status: OK
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24598</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:53:19.000Z" />
<EventRecordID>15420</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>2</Data>
<Data>RECOVERING</Data>
<Data>OK</Data>
</EventData>
</Event>

Log Name: System
Source: HP Smart Array
Date: 08/09/2010 18:53:46
Event ID: 201
Task Category: Primary Storage Device
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
A physical drive status has changed from a non-OK state to OK.

User Action
No action is recommended.

WBEM Indication Properties
AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_DiskDrive.CreationClassName="HPSA_DiskDrive",DeviceID="5001438007CBDEF0:4",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A physical drive status has changed from a non-OK state to OK."
DeviceControllerLogicalLocation[0]: "Smart Array P410i in Slot 0"
DeviceModel: "EG0146FAWHU"
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "201"
EventTime: "20100908185338.637000-000"
HWFirmwareVersion[0]: "HPDD"
HWLogicalLocation[0]: "Port:2I Box:1 Bay:5"
HWManufacturer: "HP"
HWPhysicalLocation: "Box 1 Bay 5"
HWSerialNumber[0]: "3SD35GK300009037KLQ5"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{24FDD0C6-9A89-42EC-8354-4EE07F5EF205}"
IndicationTime: "20100908185343.644000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 2 0x2 (Information)
ProbableCause: 1 0x1 (Other)
ProbableCauseDescription: "Physical Drive Status Changed to OK"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "No action is recommended."
Summary: "Physical Drive status OK"
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284420266712209 0x1cb4f7ec8fd5c91
VariableNames[0]: "FailureDescription"
VariableNames[1]: "LastFailureDescription"
VariableNames[2]: "DriveInterface"
VariableTypes[0]: 1 0x1 (string)
VariableTypes[1]: 1 0x1 (string)
VariableTypes[2]: 4 0x4 (uint16)
VariableValues[0]: "OK"
VariableValues[1]: "Hot removed"
VariableValues[2]: "3"

For more information, please contact HP Support.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="HP Smart Array" />
<EventID Qualifiers="16384">201</EventID>
<Level>4</Level>
<Task>6</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:53:46.000Z" />
<EventRecordID>15421</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>A physical drive status has changed from a non-OK state to OK.</Data>
<Data>No action is recommended.</Data>
<Data>AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_DiskDrive.CreationClassName="HPSA_DiskDrive",DeviceID="5001438007CBDEF0:4",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A physical drive status has changed from a non-OK state to OK."
DeviceControllerLogicalLocation[0]: "Smart Array P410i in Slot 0"
DeviceModel: "EG0146FAWHU"
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "201"
EventTime: "20100908185338.637000-000"
HWFirmwareVersion[0]: "HPDD"
HWLogicalLocation[0]: "Port:2I Box:1 Bay:5"
HWManufacturer: "HP"
HWPhysicalLocation: "Box 1 Bay 5"
HWSerialNumber[0]: "3SD35GK300009037KLQ5"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{24FDD0C6-9A89-42EC-8354-4EE07F5EF205}"
IndicationTime: "20100908185343.644000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 2 0x2 (Information)
ProbableCause: 1 0x1 (Other)
ProbableCauseDescription: "Physical Drive Status Changed to OK"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "No action is recommended."
Summary: "Physical Drive status OK"
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284420266712209 0x1cb4f7ec8fd5c91
VariableNames[0]: "FailureDescription"
VariableNames[1]: "LastFailureDescription"
VariableNames[2]: "DriveInterface"
VariableTypes[0]: 1 0x1 (string)
VariableTypes[1]: 1 0x1 (string)
VariableTypes[2]: 4 0x4 (uint16)
VariableValues[0]: "OK"
VariableValues[1]: "Hot removed"
VariableValues[2]: "3"
</Data>
<Data>For more information, please contact HP Support.</Data>
</EventData>
</Event>

Log Name: System
Source: HP Smart Array
Date: 08/09/2010 18:53:46
Event ID: 101
Task Category: Primary Storage Device
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
A logical drive status has changed from a non-OK state to OK.

User Action
No action is recommended.

WBEM Indication Properties
AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_StorageVolume.CreationClassName="HPSA_StorageVolume",DeviceID="5001438007CBDEF0-LD1",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A logical drive status has changed from a non-OK state to OK."
DeviceOperationalStatus[0]: 2 0x2 (OK)
DeviceOperationalStatusPrevious[0]: 3 0x3 (Degraded)
DeviceOperationalStatusPrevious[1]: 32777 0x8009 (Recovering)
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "101"
EventTime: "20100908185338.652000-000"
HWLogicalLocation[0]: "Logical Drive 2 (RAID 50)"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{BB0D2DDE-07F4-4F97-B190-437B4843AF2E}"
IndicationTime: "20100908185346.686000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 2 0x2 (Information)
ProbableCause: 1 0x1 (Other)
ProbableCauseDescription: "Logical Drive Status Changed to OK"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "No action is recommended."
Summary: "Logical drive status OK"
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284420267804211 0x1cb4f7ec90e0633
VariableNames[0]: "FailureDescription"
VariableTypes[0]: 1 0x1 (string)
VariableValues[0]: "N/A"

For more information, please contact HP Support.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="HP Smart Array" />
<EventID Qualifiers="16384">101</EventID>
<Level>4</Level>
<Task>6</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T17:53:46.000Z" />
<EventRecordID>15422</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>A logical drive status has changed from a non-OK state to OK.</Data>
<Data>No action is recommended.</Data>
<Data>AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\HWEDHPV\ROOT\HPQ:HPSA_StorageVolume.CreationClassName="HPSA_StorageVolume",DeviceID="5001438007CBDEF0-LD1",SystemCreationClassName="HPSA_ArraySystem",SystemName="5001438007CBDEF0""
AlertType: 5 0x5 (Device Alert)
Description: "A logical drive status has changed from a non-OK state to OK."
DeviceOperationalStatus[0]: 2 0x2 (OK)
DeviceOperationalStatusPrevious[0]: 3 0x3 (Degraded)
DeviceOperationalStatusPrevious[1]: 32777 0x8009 (Recovering)
EventCategory: 6 0x6 (Primary Storage Device)
EventID: "101"
EventTime: "20100908185338.652000-000"
HWLogicalLocation[0]: "Logical Drive 2 (RAID 50)"
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{BB0D2DDE-07F4-4F97-B190-437B4843AF2E}"
IndicationTime: "20100908185346.686000+060"
NetworkAddresses[0]: "fe80::c17c:eb2d:89dc:a1bd%1034"
NetworkAddresses[1]: "192.168.5.16"
NetworkAddresses[2]: "fe80::44ef:3fa7:5928:daed%1033"
NetworkAddresses[3]: "169.254.48.111"
NetworkAddresses[4]: "fe80::34b7:b748:56c4:953c%12"
NetworkAddresses[5]: "192.168.5.4"
OSType: 103 0x67 (Microsoft Windows Server 2008 R2)
OSVersion: "6.1.7600"
PerceivedSeverity: 2 0x2 (Information)
ProbableCause: 1 0x1 (Other)
ProbableCauseDescription: "Logical Drive Status Changed to OK"
ProviderName: "HP Smart Array"
ProviderVersion: "2.5.0.0"
RecommendedActions[0]: "No action is recommended."
Summary: "Logical drive status OK"
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2010.03.01"
SystemFirmwareVersion[1]: "2010.03.01"
SystemGUID: "30303734-3536-5A43-4A30-313230364531"
SystemModel: "ProLiant ML350 G6"
SystemName: "HWEDHPV"
SystemProductID: "470065-164"
SystemSerialNumber: "CZJ01206E1"
TIME_CREATED: 129284420267804211 0x1cb4f7ec90e0633
VariableNames[0]: "FailureDescription"
VariableTypes[0]: 1 0x1 (string)
VariableValues[0]: "N/A"
</Data>
<Data>For more information, please contact HP Support.</Data>
</EventData>
</Event>

Log Name: System
Source: Cissesrv
Date: 08/09/2010 21:42:54
Event ID: 24684
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: HWEDHPV
Description:
The consistency initialization pass has completed for logical drive 1 of array controller P410i [Embedded].
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cissesrv" />
<EventID Qualifiers="17461">24684</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-09-08T20:42:54.000Z" />
<EventRecordID>15423</EventRecordID>
<Channel>System</Channel>
<Computer>HWEDHPV</Computer>
<Security />
</System>
<EventData>
<Data>Cissesrv</Data>
<Data>P410i [Embedded]</Data>
<Data>1</Data>
</EventData>
</Event>

Member_2_231077

Not reading through all that, is it always bay 5? It'll be a lot easier to see using the Systems Management Homepage integrated log viewer as that lists all hardware related problems.

First thing to do is to upgrade the firmware on both the controller and the disk drives to the latest version.

solplus

ASKER

Thanks andyalder.

Will look into the bay query and get back to you.
The server is running Hyper V Server 2008 and doesnt have a gui. Can the Systems Management homepage be installed from a command line prompt? If so any guidance on how to do it would be much appreciated.

Member_2_231077

SMH should be installed if you did a smartstart install, try opening a webpage to http://<server IP>:2301 or HTTPS://<serverIP>:2381

Haven't tried installing from command line but if you download the Proliant Support Pack you can do a push install from a remote PC.

solplus

ASKER

Thanks andyalder.
We havent done a push install before. How do we do it?

Member_2_231077

It's pretty self evident, run the prog on a remote PC and there's option to put remote name/address in and click green arrow. It won't install stuff that's not relevant but it may cause a reboot or two if it loads newer drivers so don't do in business hours.

solplus

ASKER

Logged a call with HP. They want me to run the Array diagnostics. That means shutting down the server and booting from the CD. The problem is that we only notived the errors in the event viewer when we were investigating why backups were going slow and failing, so we do not have a good backup of the server and are a bit reticent to shutdown in this situation in case it does not come back up. At least at the moment it is working. That is aslo why we have not followed up your suggestion of installing the Systems Management Homepage, as you thought it might cause a reboot. Currenty backing up the virtual servers and virtual discs to an external disc and will look for a slot to down the server after that has completed. I really cannot see why the Array Diagnostics have to be run at boot time from the CD. I would have though that there should be an online tool that can be used to interogate the array?

solplus

ASKER

Sorry, to answer your earlier question. Yes it is alway bay 5 and it would appear to have been happening on and of for a few weeks. The problem is that removing/replacing a disc is in theory legitamate, so the removal and reinstallation and not reported as errors, or even warnings, but are reported as informational and get lost in the 1000's of other informational events.

ASKER CERTIFIED SOLUTION

solplus

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

solplus

ASKER

If the event log says that a disc is being removed and you know that it is not, then it has to be a problem with the software, the bay or the disc. If it always reports the disc being removed from the same bay, then the problem has to be the bay or the disc. As the disc is easier to replace than the bay, I suggest trying a replacement disc to eliminate this. If you do not have a replacement disc then log a hardware fault and grit your teeth until they are fed up running diagnostices.