CSV will only come online on 1 of 3 hosts after power loss

3 node Hyper V 2008 R2
Dell Equallogic SAN using iSCSI
1 CSV + Quorum

Yesterday we had an issue with our AV that hung the 3 hosts. The VMs continued to run but we could not access the hosts. I ended up shutting down all VMs through remote desktop then hard power off each of the hosts.

Powered the hosts back on and all VMs came up fine but now the CSV will only come online on 1 host. I checked SAN connectivity on all hosts and all can access communicate with the SAN.

I have rebooted the 2 nodes that cannot bring the CSV online, but no change.

Event viewer on both nodes that fail have the below errors

1205
The Cluster service failed to bring clustered service or application '510cb678-1339-4407-b895-b99330eaa883' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.

1069
Cluster resource 'Cluster Disk 2' in clustered service or application '510cb678-1339-4407-b895-b99330eaa883' failed.

Below is the cluster log from one of the nodes that the CSV failed to move and come online

5/06/24-13:13:50.794 INFO  [DCM] Processing message dcm/pause
0000096c.00000bc8::2015/06/24-13:13:50.794 INFO  [DCM] Push.AsyncPauseDisk for 3945a7d4-69cd-4c77-b504-6f243b816c1e
0000096c.0000111c::2015/06/24-13:13:50.794 INFO  [DCM] SyncHandler for 3945a7d4-69cd-4c77-b504-6f243b816c1e
0000096c.0000111c::2015/06/24-13:13:50.794 INFO  [DCM] enter_AllGood(3945a7d4-69cd-4c77-b504-6f243b816c1e) P0..75 P0..150
0000096c.0000111c::2015/06/24-13:13:50.794 INFO  [DCM] MappingManager::PauseVolume 'Volume1'
0000096c.0000111c::2015/06/24-13:13:50.794 INFO  [DCM] Filter.ChangeState (ctx=2, state=CfsVolumeStatePaused)
0000096c.000011a0::2015/06/24-13:13:50.919 WARN  [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to Cluster Disk 2 returned 5004.
00000e7c.00001bf0::2015/06/24-13:13:51.278 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.278 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001bf0::2015/06/24-13:13:51.293 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.293 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.309 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000011a0::2015/06/24-13:13:51.309 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
0000096c.000011a0::2015/06/24-13:13:51.309 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
00000e7c.000013c0::2015/06/24-13:13:51.309 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.309 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.309 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.000011a0::2015/06/24-13:13:51.309 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.000011a0::2015/06/24-13:13:51.309 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
00000e7c.000013c0::2015/06/24-13:13:51.309 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.309 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.324 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.324 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001bf0::2015/06/24-13:13:51.340 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.340 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.948 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000011a0::2015/06/24-13:13:51.948 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001bf0::2015/06/24-13:13:51.964 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000e7c.000013c0::2015/06/24-13:13:51.964 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.964 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
0000096c.00001d18::2015/06/24-13:13:51.964 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
0000096c.000011a0::2015/06/24-13:13:51.964 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.964 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000011a0::2015/06/24-13:13:51.964 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.964 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d18::2015/06/24-13:13:51.964 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.00001d18::2015/06/24-13:13:51.964 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
00000e7c.000013c0::2015/06/24-13:13:51.964 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.000011a0::2015/06/24-13:13:51.964 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.980 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000011a0::2015/06/24-13:13:51.980 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:13:51.995 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:13:51.995 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.00001d18::2015/06/24-13:13:52.276 WARN  [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to Cluster Disk 2 returned 5004.
0000096c.00000c48::2015/06/24-13:13:54.819 INFO  [DCM] filter.Event ->CfsVolumeStatePaused FromPause for ctx=2 status 00000000
0000096c.0000111c::2015/06/24-13:13:54.819 INFO  [DCM] volume paused 'Volume1'
0000096c.00000c50::2015/06/24-13:13:54.819 INFO  [GUM] Node 2: Processing RequestLock 3:1097
0000096c.00000c34::2015/06/24-13:13:54.819 INFO  [GUM] Node 2: Processing GrantLock to 3 (sent by 1 gumid: 2567)
0000096c.00001064::2015/06/24-13:13:55.677 ERR   [RCM] s_RcmRpcGetResourceState: ERROR_CLUSTER_GROUP_MOVING(5908)' because of ''Cluster Disk 2' is owned by node 3, not 2.'
0000096c.00001d18::2015/06/24-13:13:55.677 WARN  [RCM] rcm::RcmApi::ResourceControl: forwarded, no retry on error 5908
0000096c.00001d18::2015/06/24-13:13:55.677 WARN  [RCM] ResourceControl(GET_CLASS_INFO) to Cluster Disk 2 returned 5908.
0000096c.000011a0::2015/06/24-13:13:55.677 WARN  [RCM] rcm::RcmApi::ResourceControl: retrying: 510cb678-1339-4407-b895-b99330eaa883, 5908.
0000096c.00001064::2015/06/24-13:13:55.677 WARN  [RCM] rcm::RcmApi::GetResourceState: retrying: 510cb678-1339-4407-b895-b99330eaa883, 5908.
0000096c.00000ae8::2015/06/24-13:13:55.677 INFO  [RCM] rcm::RcmGum::GroupMoveOperation(510cb678-1339-4407-b895-b99330eaa883,2)
0000096c.00001d18::2015/06/24-13:13:55.677 INFO  [RCM] TransitionToState(Cluster Disk 2) Offline-->OnlineCallIssued.
0000096c.00001d18::2015/06/24-13:13:55.677 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (510cb678-1339-4407-b895-b99330eaa883, BringingToPersistentState --> Pending)
000012f0.00001dac::2015/06/24-13:13:55.677 INFO  [RES] Physical Disk <Cluster Disk 2>: Online request.
0000096c.000018c0::2015/06/24-13:13:55.677 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'Cluster Disk 2', gen(3) result 997.
0000096c.000018c0::2015/06/24-13:13:55.677 INFO  [RCM] TransitionToState(Cluster Disk 2) OnlineCallIssued-->OnlinePending.
000012f0.00001e74::2015/06/24-13:13:55.677 INFO  [RES] Physical Disk <Cluster Disk 2>: Arbitrate request: FastPath 0
000012f0.00001e74::2015/06/24-13:13:57.783 INFO  [RES] Physical Disk: Successful reserve, disk is unowned, key 7cb77466734d
000012f0.00001e74::2015/06/24-13:13:57.783 INFO  [RES] Physical Disk: Update disk props returns 0
000012f0.00001e74::2015/06/24-13:13:57.798 INFO  [RES] Physical Disk <Cluster Disk 2>: Disk is offline
0000096c.00001d18::2015/06/24-13:13:57.798 INFO  [GUM] Node 2: Processing RequestLock 2:136
0000096c.00000c50::2015/06/24-13:13:57.798 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 3 gumid: 2569)
000012f0.00001e74::2015/06/24-13:13:57.798 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00000a04::2015/06/24-13:13:57.798 INFO  [RES] Physical Disk <Cluster Disk 2>: Completed HardDiskpUpdateRegistryWorker (start time 13:13:57.798)
000012f0.00001e74::2015/06/24-13:14:01.808 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:05.817 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:09.826 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:13.835 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:17.844 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:21.853 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
0000096c.000018c0::2015/06/24-13:14:23.382 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_NOT_ONLINE(5004)' because of 'PreprocessControl(16777765) failed for resource 'Cluster Disk 2'.'
0000096c.000018c0::2015/06/24-13:14:23.382 WARN  [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to Cluster Disk 2 returned 5004.
0000096c.00001d18::2015/06/24-13:14:23.413 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_NOT_ONLINE(5004)' because of 'PreprocessControl(16777765) failed for resource 'Cluster Disk 2'.'
0000096c.00001d18::2015/06/24-13:14:23.413 WARN  [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to Cluster Disk 2 returned 5004.
0000096c.00001064::2015/06/24-13:14:23.429 ERR   [RCM] rcm::RcmResControl::DoResourceControl: ERROR_RESOURCE_NOT_ONLINE(5004)' because of 'PreprocessControl(16777765) failed for resource 'Cluster Disk 2'.'
0000096c.00001064::2015/06/24-13:14:23.429 WARN  [RCM] ResourceControl(STORAGE_GET_SHARED_VOLUME_INFO) to Cluster Disk 2 returned 5004.
00000e7c.000013c0::2015/06/24-13:14:23.756 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d18::2015/06/24-13:14:23.756 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:14:23.772 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d18::2015/06/24-13:14:23.772 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001d54::2015/06/24-13:14:23.788 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:14:23.788 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
0000096c.00001064::2015/06/24-13:14:23.788 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
00000e7c.00001d54::2015/06/24-13:14:23.788 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d18::2015/06/24-13:14:23.788 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001d54::2015/06/24-13:14:23.788 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:14:23.788 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.00001064::2015/06/24-13:14:23.788 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
00000e7c.00001d54::2015/06/24-13:14:23.788 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d18::2015/06/24-13:14:23.788 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001d54::2015/06/24-13:14:23.803 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d18::2015/06/24-13:14:23.803 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:14:23.819 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:14:23.819 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
000012f0.00001d70::2015/06/24-13:14:24.256 ERR   [RES] Physical Disk: Failed to get partition size, status 3
000012f0.00001dac::2015/06/24-13:14:24.287 ERR   [RES] Physical Disk: Failed to get partition size, status 3
000012f0.00001d70::2015/06/24-13:14:24.302 ERR   [RES] Physical Disk: Failed to get partition size, status 3
000012f0.00001e74::2015/06/24-13:14:25.862 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:29.872 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:33.881 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:37.890 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:41.899 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:45.908 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:49.917 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:53.926 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
00000e7c.00001338::2015/06/24-13:14:56.547 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d6c::2015/06/24-13:14:56.547 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:14:56.859 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d6c::2015/06/24-13:14:56.859 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:14:56.875 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d6c::2015/06/24-13:14:56.875 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:14:56.875 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d6c::2015/06/24-13:14:56.875 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
000012f0.00001e74::2015/06/24-13:14:57.936 WARN  [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Failed to get volume guid for device \\?\GLOBALROOT\Device\Harddisk3\Partition1\. Error 3
000012f0.00001e74::2015/06/24-13:14:57.936 ERR   [RES] Physical Disk <Cluster Disk 2>: OnlineThread: No volume arrival in 60 seconds, failing online.
000012f0.00001e74::2015/06/24-13:14:57.936 ERR   [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Error 3 bringing resource online.
000012f0.00001e74::2015/06/24-13:14:58.341 INFO  [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 3 (sig 37561d87)
000012f0.00001e74::2015/06/24-13:14:58.372 INFO  [RES] Physical Disk <Cluster Disk 2>: HardDiskpCloseSVIHandles: Exit
000012f0.00001e74::2015/06/24-13:14:58.372 WARN  [RHS] Resource Cluster Disk 2 has indicated that it cannot come online on this node.
0000096c.00001d6c::2015/06/24-13:14:58.372 WARN  [RCM] HandleMonitorReply: ONLINERESOURCE for 'Cluster Disk 2', gen(3) result 5015.
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] TransitionToState(Cluster Disk 2) OnlinePending-->ProcessingCannotComeOnline.
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (510cb678-1339-4407-b895-b99330eaa883, Pending --> Failed)
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] Group move for '510cb678-1339-4407-b895-b99330eaa883' has completed.
0000096c.00001d6c::2015/06/24-13:14:58.372 ERR   [RCM] rcm::RcmResource::HandleFailure: (Cluster Disk 2)
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [DCM] ResourceFailure for Cluster Disk 2
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] resource Cluster Disk 2: failure count: 3, restartAction: 2.
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] Greater than restartPeriod time has elapsed since first failure, resetting failureTime and failureCount.
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] TransitionToState(Cluster Disk 2) ProcessingCannotComeOnline-->[WaitingToTerminate to CannotComeOnlineOnThisNode].
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (510cb678-1339-4407-b895-b99330eaa883, Failed --> Pending)
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] TransitionToState(Cluster Disk 2) [WaitingToTerminate to CannotComeOnlineOnThisNode]-->[Terminating to CannotComeOnlineOnThisNode].
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] Resource Cluster Disk 2 is causing group 510cb678-1339-4407-b895-b99330eaa883 to failover.  Posting worker thread.
000012f0.00001d70::2015/06/24-13:14:58.372 INFO  [RES] Physical Disk <Cluster Disk 2>: Terminate request.
000012f0.00001d70::2015/06/24-13:14:58.372 WARN  [RES] Physical Disk <Cluster Disk 2>: Terminate: Failed to open device \Device\Harddisk3\Partition1, Error 2
0000096c.00001d6c::2015/06/24-13:14:58.372 INFO  [RCM] rcm::RcmGroup::Failover: (510cb678-1339-4407-b895-b99330eaa883)
000012f0.00001d70::2015/06/24-13:14:58.372 INFO  [RES] Physical Disk <Cluster Disk 2>: HardDiskpCloseSVIHandles: Exit
0000096c.00001324::2015/06/24-13:14:58.372 INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'Cluster Disk 2', gen(4) result 0.
0000096c.00001324::2015/06/24-13:14:58.372 INFO  [RCM] TransitionToState(Cluster Disk 2) [Terminating to CannotComeOnlineOnThisNode]-->CannotComeOnlineOnThisNode.
0000096c.00001324::2015/06/24-13:14:58.372 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (510cb678-1339-4407-b895-b99330eaa883, Pending --> Failed)
0000096c.00001d6c::2015/06/24-13:14:58.404 INFO  [RCM] time since last failure is greater than failover period; resetting failoverCount to 1.
0000096c.00001d6c::2015/06/24-13:14:58.404 WARN  [RCM] Failing over group 510cb678-1339-4407-b895-b99330eaa883, failoverCount 1, last time 2015/06/24-08:14:58.404.
0000096c.00001d6c::2015/06/24-13:14:58.404 INFO  [RCM] rcm::RcmGroup::Move: (510cb678-1339-4407-b895-b99330eaa883, 3)
0000096c.00001d6c::2015/06/24-13:14:58.404 INFO  [RCM] No need to bring group '510cb678-1339-4407-b895-b99330eaa883' offline before move; state Failed, owner 2.
0000096c.00001d6c::2015/06/24-13:14:58.404 INFO  [RCM] rcm::RcmGum::GroupMoveOperation(510cb678-1339-4407-b895-b99330eaa883,3)
0000096c.00000c50::2015/06/24-13:14:59.464 INFO  [GUM] Node 2: Processing RequestLock 3:1099
0000096c.00000c50::2015/06/24-13:14:59.464 INFO  [GUM] Node 2: Processing GrantLock to 3 (sent by 2 gumid: 2571)
0000096c.00000bc8::2015/06/24-13:15:01.570 INFO  [DCM] Processing message dcm/pause
0000096c.00000bc8::2015/06/24-13:15:01.570 INFO  [DCM] Push.AsyncPauseDisk for 3945a7d4-69cd-4c77-b504-6f243b816c1e
0000096c.0000111c::2015/06/24-13:15:01.570 INFO  [DCM] SyncHandler for 3945a7d4-69cd-4c77-b504-6f243b816c1e
0000096c.0000111c::2015/06/24-13:15:01.570 INFO  [DCM] enter_AllGood(3945a7d4-69cd-4c77-b504-6f243b816c1e) P0..75 P0..150
0000096c.0000111c::2015/06/24-13:15:01.570 INFO  [DCM] volume 'Volume1' is already paused
0000096c.00000bc8::2015/06/24-13:15:02.054 INFO  [DCM] Processing message dcm/map
0000096c.00000bc8::2015/06/24-13:15:02.054 INFO  [DCM] Push.AsyncMapDisk for 3945a7d4-69cd-4c77-b504-6f243b816c1e
0000096c.0000111c::2015/06/24-13:15:02.054 INFO  [DCM] SyncHandler for 3945a7d4-69cd-4c77-b504-6f243b816c1e
0000096c.0000111c::2015/06/24-13:15:02.054 INFO  (localInterface->Addresses, network->Prefixes, interface1Network) = (remoteInterface->Addresses, network->Prefixes, interface2Network ) = (finalList) = [DCM] Reservation.SetMembership(Cluster Disk 2,(1 2 3))
000012f0.00001f64::2015/06/24-13:15:02.054 ERR   [RHS] Error 5023 from ResourceControl for resource Cluster Disk 2.
0000096c.0000111c::2015/06/24-13:15:02.054 INFO  [DCM] Reservation.SetPrKey(Cluster Disk 2,2)
000012f0.00001d70::2015/06/24-13:15:02.054 WARN  [RES] Physical Disk <Cluster Disk 2>: SetSharedPRKey: Setting shared PR key on disk that is not Offline on this node.
000012f0.00001d70::2015/06/24-13:15:02.085 INFO  [RES] Physical Disk <Cluster Disk 2>: SetSharedPRKey: registered shared PR key high 00004B71 low 0002734D for device 3
000012f0.00001d70::2015/06/24-13:15:02.101 INFO  [RES] Physical Disk: FindTheDiskNumber: Cleared log once flag, disk 3, 3, signature 0x37561d87
0000096c.0000111c::2015/06/24-13:15:02.148 INFO  [DCM] MapDisk mapping volume 'Volume1'
0000096c.0000111c::2015/06/24-13:15:02.148 INFO  [DCM] volume 'Volume1' is already paused
0000096c.0000111c::2015/06/24-13:15:02.148 INFO  [DCM] CreateLink C:\ClusterStorage\Volume1 => \\10.100.1.10\3945a7d4-69cd-4c77-b504-6f243b816c1e-1048576$\
0000096c.0000111c::2015/06/24-13:15:02.148 INFO  [DCM] tag is already set for C:\ClusterStorage\Volume1\
0000096c.0000111c::2015/06/24-13:15:02.148 INFO  [DCM] DcmCreateLink 'C:\ClusterStorage\Volume1\': link is already there
0000096c.0000111c::2015/06/24-13:15:02.148 INFO  [DCM] Filter.Activate (ctx=2,isLocal=0,DiskNo=3,512,00000000,00100000,\\10.100.1.10\3945a7d4-69cd-4c77-b504-6f243b816c1e-1048576$\)
0000096c.00000c48::2015/06/24-13:15:02.319 INFO  [DCM] filter.Event ->CfsVolumeStateActive FromActivate for ctx=2 status 00000000
0000096c.00000c48::2015/06/24-13:15:02.319 INFO  [DCM] enter_CountingToGood(3945a7d4-69cd-4c77-b504-6f243b816c1e) R1..75 P0..150
0000096c.0000111c::2015/06/24-13:15:02.319 INFO  [DCM] (VolumeWatcher) StartWatcher(enter)
0000096c.0000111c::2015/06/24-13:15:02.319 INFO  [DCM] (VolumeWatcher) StartWatcher(exit): threadid 3012
0000096c.0000111c::2015/06/24-13:15:02.319 INFO  [DCM] (VolumeWatcher) Added volume symlink Volume1 to watch list
0000096c.0000111c::2015/06/24-13:15:02.319 INFO  [DCM] MapDisk cleared log once flag, vol 3945a7d4-69cd-4c77-b504-6f243b816c1e:1048576
00000e7c.00001338::2015/06/24-13:15:02.335 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d04::2015/06/24-13:15:02.335 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:02.350 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d04::2015/06/24-13:15:02.350 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:02.397 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000010d8::2015/06/24-13:15:02.397 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
0000096c.000010d8::2015/06/24-13:15:02.397 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
00000e7c.00001338::2015/06/24-13:15:02.397 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d6c::2015/06/24-13:15:02.397 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:02.397 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.000010d8::2015/06/24-13:15:02.397 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.000010d8::2015/06/24-13:15:02.397 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
00000e7c.00001338::2015/06/24-13:15:02.397 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d6c::2015/06/24-13:15:02.397 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.00001d6c::2015/06/24-13:15:02.787 INFO  [DCM] SubmitPropertyChangeNotification: resourceId 3945a7d4-69cd-4c77-b504-6f243b816c1e
00000e7c.00001d54::2015/06/24-13:15:02.990 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:02.990 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:03.006 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:15:03.006 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:03.068 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d04::2015/06/24-13:15:03.068 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
0000096c.00001d04::2015/06/24-13:15:03.068 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
00000e7c.00001338::2015/06/24-13:15:03.068 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:03.068 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:03.068 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d04::2015/06/24-13:15:03.068 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.00001d04::2015/06/24-13:15:03.068 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
00000e7c.00001338::2015/06/24-13:15:03.068 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:15:03.068 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.000018f4::2015/06/24-13:15:03.583 INFO  [NM] Received request from client address fe80::6949:107a:bd2d:a721.
0000096c.000018f4::2015/06/24-13:15:03.630 WARN  Resource type Microsoft Exchange Information Store not found.
0000096c.00001064::2015/06/24-13:15:03.864 INFO  [DCM] SubmitPropertyChangeNotification: resourceId 3945a7d4-69cd-4c77-b504-6f243b816c1e
00000e7c.00001338::2015/06/24-13:15:03.895 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d04::2015/06/24-13:15:03.895 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:03.910 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d04::2015/06/24-13:15:03.910 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:04.550 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:04.550 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:04.566 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.000018f4::2015/06/24-13:15:04.566 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001d54::2015/06/24-13:15:04.924 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d04::2015/06/24-13:15:04.924 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:04.940 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:04.940 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001d54::2015/06/24-13:15:04.940 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:15:04.940 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
0000096c.00001d04::2015/06/24-13:15:04.940 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001d54::2015/06/24-13:15:04.940 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d04::2015/06/24-13:15:04.940 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001d54::2015/06/24-13:15:04.940 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:15:04.940 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.00001064::2015/06/24-13:15:04.940 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
00000e7c.00001d54::2015/06/24-13:15:04.940 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d04::2015/06/24-13:15:04.940 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:05.533 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:05.533 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:05.548 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:15:05.548 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:05.564 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:05.564 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:05.564 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:15:05.564 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:05.579 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:05.579 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:05.595 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000018f4::2015/06/24-13:15:05.595 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:15:05.595 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.000018f4::2015/06/24-13:15:05.595 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
0000096c.00001064::2015/06/24-13:15:05.595 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:15:05.595 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001064::2015/06/24-13:15:05.595 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:15:05.595 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.000018f4::2015/06/24-13:15:05.595 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.000018f4::2015/06/24-13:15:05.595 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
00000e7c.000013c0::2015/06/24-13:15:05.595 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001064::2015/06/24-13:15:05.595 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
0000096c.00000c34::2015/06/24-13:15:05.891 INFO  [GUM] Node 2: Processing RequestLock 1:853
0000096c.00000c50::2015/06/24-13:15:05.891 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 3 gumid: 2573)
0000096c.00001d6c::2015/06/24-13:15:06.375 INFO  [NM] Received request from client address fe80::6949:107a:bd2d:a721.
00000e7c.00001338::2015/06/24-13:15:06.531 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d6c::2015/06/24-13:15:06.531 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:06.547 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d6c::2015/06/24-13:15:06.547 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:07.186 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000018f4::2015/06/24-13:15:07.186 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:07.202 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.000018f4::2015/06/24-13:15:07.202 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.00001338::2015/06/24-13:15:16.453 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000018f4::2015/06/24-13:15:16.453 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.00001338::2015/06/24-13:15:16.453 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.000018f4::2015/06/24-13:15:16.453 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:15:16.453 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d6c::2015/06/24-13:15:16.453 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:15:16.468 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d6c::2015/06/24-13:15:16.468 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000e7c.000013c0::2015/06/24-13:15:16.468 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
0000096c.00001d6c::2015/06/24-13:15:16.468 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000e7c.000013c0::2015/06/24-13:15:16.468 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
0000096c.00001d6c::2015/06/24-13:15:16.468 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.

Open in new window

jwcchelpdeskAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Make sure the node that is online is owner of _all_ resources. Please snip and post a shot of FCM's storage node and Role node.
jwcchelpdeskAuthor Commented:
Capture.PNG
Storage screen attached. I am not sure what you mean by "role node"

All nodes online.

How can I check if node is owner of the storage? It does not appear in the properties of the CSV or anywhere else I can think to look.
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Current Owner = red

At the root of FCM is the cluster name. Click on that and snip it to here please. Make sure to show Cluster Name and IP address nodes with their status.
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

jwcchelpdeskAuthor Commented:
So I found that on the 2 with issues there was a device in device manager with a question mark next to it. Not yellow. I uninstalled and rebooted, then they were fine.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
What was the device?
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
What was the device?
jwcchelpdeskAuthor Commented:
I don't remember as this was nearly 2 months ago
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Suggestion: Keep an audit of everything for both KB and change control. Makes it easier to figure out what is going on if the same situation happens again. :)
jwcchelpdeskAuthor Commented:
It is correct
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Virtualization

From novice to tech pro — start learning today.