Solved

Cluster Services not Mounting HDD's properly?

Posted on 2010-08-16
7
1,059 Views
Last Modified: 2012-05-10


Good Morning folks, hope someone has an idea on this one because i'm stumped!

We've got a cluster running for our exchange servers, and at the moment it's running fine with node1 handling all the work load.  However when we try to test the fail over of the databases to node2 we run into a bit of an issue.

Once we take node1 offline, node2 cluster services SEEM to start spinning up.
It's broken intto two parts.

First part includes the following:
Cluster IP
Cluster Name
Exchange HDD mapings (E:,Q:,L:,S)
MSDTC

Part two however is where we seem to be running into issues (tho I suspect an earlier probelm in the sequence).

Part 2 as follows:
Exchange Information Store Instance
Exchange system Attendant Instance
Exchange Storage Group - Retired
Exchange Storage Group - Bes
Exhange Storage Group - Primary Storage Database

IP address
Network Name

The 3 bolded items are the ones that show up as failed in the management window.
However when I check the Win Explorer, I don't actually see the drive mappings from Phase 1 being loaded, ie there's no E/Q/L/S drives being displayed, which obviously would kill the concept of the physcal disks being mounted.

Example error in Eventvwr:
Event ID: 1034
The disk associated with cluster disk resource 'Disk Q: L:' could not be found. The expected signature of the disk was 28D73749. If the disk was removed from the server cluster, the resource should be deleted. If the disk was replaced, the resource must be deleted and created again in order to bring the disk online. If the disk has not been removed or replaced, it may be inaccessible at this time because it is reserved by another server cluster node.


Also Attache is a bunch of error information from the cluster error services.

Perhaps someone with a bigger brain then me can decipher?

Thanks,
Post
adc:af8.08/04[09:17:56.375](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:17:56.375](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:17:56.375](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 1296 for 'DwTerminateThread'.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '7c8edd60-c45c-44de-ac2b-3751677563f9' from resource.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property 'e75836bb-6f55-46d3-8060-879bbe618e65' from resource.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16
adc:510.08/04[09:17:56.375](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1
adc:510.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EventLogging: First Storage Group/Mailbox Database (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed.  Error Code: 1.
adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.
adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.
adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 612 for 'DwTerminateThread'.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '7c8edd60-c45c-44de-ac2b-3751677563f9' from resource.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property '7afce291-6cb7-4648-aa85-400eeaffb36b' from resource.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16
adc:264.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1
adc:264.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EventLogging: First Storage Group/bes (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed.  Error Code: 1.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 1368 for 'DwTerminateThread'.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '8a1a2af4-860f-4510-8ff7-2cd071b8b045' from resource.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property '46243d7f-fd85-4aea-b787-71f90ae585b7' from resource.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16
adc:558.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1
adc:558.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EventLogging: ExchangeBack/Retired Users (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed.  Error Code: 1.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:578.08/04[09:18:08.609](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:18:08.609](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:18:08.609](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:08.609](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:d08.08/04[09:18:20.781](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:18:20.781](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:18:20.781](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:20.781](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af4.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af4.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:d28.08/04[09:18:32.937](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:18:32.937](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:18:32.937](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:32.937](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:18:32.937](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:b4c.08/04[09:18:32.937](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:f44.08/04[09:19:01.781](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:19:01.781](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:19:01.781](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:01.781](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:19:01.781](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:f34.08/04[09:19:01.781](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:f88.08/04[09:19:30.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:19:30.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:19:30.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:30.796](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:19:30.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:f7c.08/04[09:19:30.796](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:fc4.08/04[09:19:59.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:19:59.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:19:59.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:59.796](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:19:59.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:fb8.08/04[09:19:59.796](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:3d4.08/04[09:20:28.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:20:28.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:20:28.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:20:28.796](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:20:28.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:0c4.08/04[09:20:28.796](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
974:980.08/12[09:02:31.571](000000) WARN [INIT] The cluster service is shutting down.

Open in new window

0
Comment
Question by:Posthumous
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
7 Comments
 
LVL 14

Expert Comment

by:Shabarinath Ramadasan
ID: 33445455
From the node which is having this issue, Can you check if all the disks mentioned are available on the Disk Manager? If you have multipath software (HDLM or Powerpath), Check the status on that too.

Go to Disk Manager and click on Rescan- This will do a scan and all the connected disks will be showed.
Make a comparison using the Size and LUN between two nodes.

Also, I would suggest you to try bringing everything offline -> Failover -> take the disk resource online one by one and see how it goes.

Good luck.
Shaba
0
 
LVL 4

Expert Comment

by:sivanov
ID: 33445497
hello ,
you forget to give as the OS of the servers :-), but since the area is win2003 i suppose they are win2k3:
i do not thing that you need someone with more brain, just more wrinkled.
What about this KB
http://support.microsoft.com/kb/280425 - an old but pretty usefull one - did you check it?
or this three - http://support.microsoft.com/kb/243195
and
http://support.microsoft.com/kb/886800.
and http://support.microsoft.com/kb/293778
0
 
LVL 14

Expert Comment

by:Shabarinath Ramadasan
ID: 33445502
I would also suggest you to bring both servers down, wait for a min and then start just the problamatic node. For saferside, Dismount all DBs and check for do not mount at startup before shutting down.
Then see how the cluster behaves.

Good luck
Shaba
0
Does Powershell have you tied up in knots?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

 
LVL 1

Author Comment

by:Posthumous
ID: 33445511

Good Morning!

When I try to fail over the cluster to node2, the cluster mgr seems to time out completely on that unit.
On Node1 (working node) I can see the listed activity above happening on the node2 but eventually the 3 items listed show as failed.  I believe (and I can't verify this until later after working hours) that the Drive letters are still visable on node1 but not accessable on node1 while this is all going on.  Eventually I have to move all the items back over to node1 to get access to the drives again and bring the exchange folders back online.

From what I can see, the disks never become available on Node2 for the exchange folders to be accessed (which is I believe the root of the issue).
0
 
LVL 1

Author Comment

by:Posthumous
ID: 33445523
Yes this is a server2003 cluster.
Sorry should have mentioned it.
0
 
LVL 14

Expert Comment

by:Shabarinath Ramadasan
ID: 33447783
Thanks for describing more on this.
Once the failover happens from Node 2 to Node1, Node2 should release the disk ownership and Node1 should get it. If this process failes, Node1 wont make the disk resource online.

The best option is to shutdown both nodes. Wait for a min. Start the problematic node and see if it helps.
Also, make sure you dismount the DBs and check "Dont mount this DB at startup" to safegaurd the DB.

Good Luck
Shaba
0
 
LVL 1

Accepted Solution

by:
Posthumous earned 0 total points
ID: 33501868
Problem turned out to be a bad Iscsi interface driver on the cluster, the driver wasn't allowing the storage device to populate properly onto the server.  The drivers were showing as connected in the device manager however no access was going through.

Removing the drivers and reinstalling the latest version from the vendor along with 2 software patches to the management software, 3 reboots and a phone call to the vendor managed to sort it out.

Thanks for the advice all.
0

Featured Post

Office 365 Training for IT Pros

Learn how to provision tenants, synchronize on-premise Active Directory, implement Single Sign-On, customize Office deployment, and protect your organization with eDiscovery and DLP policies.  Only from Platform Scholar.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article explains how to install and use the NTBackup utility that comes with Windows Server.
In-place Upgrading Dirsync to Azure AD Connect
In this video we show how to create a Contact in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Contact ta…
In this Micro Video tutorial you will learn the basics about Database Availability Groups and How to configure one using a live Exchange Server Environment. The video tutorial explains the basics of the Exchange server Database Availability grou…
Suggested Courses

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question