Solved

Cluster Services not Mounting HDD's properly?

Posted on 2010-08-16
7
1,038 Views
Last Modified: 2012-05-10


Good Morning folks, hope someone has an idea on this one because i'm stumped!

We've got a cluster running for our exchange servers, and at the moment it's running fine with node1 handling all the work load.  However when we try to test the fail over of the databases to node2 we run into a bit of an issue.

Once we take node1 offline, node2 cluster services SEEM to start spinning up.
It's broken intto two parts.

First part includes the following:
Cluster IP
Cluster Name
Exchange HDD mapings (E:,Q:,L:,S)
MSDTC

Part two however is where we seem to be running into issues (tho I suspect an earlier probelm in the sequence).

Part 2 as follows:
Exchange Information Store Instance
Exchange system Attendant Instance
Exchange Storage Group - Retired
Exchange Storage Group - Bes
Exhange Storage Group - Primary Storage Database

IP address
Network Name

The 3 bolded items are the ones that show up as failed in the management window.
However when I check the Win Explorer, I don't actually see the drive mappings from Phase 1 being loaded, ie there's no E/Q/L/S drives being displayed, which obviously would kill the concept of the physcal disks being mounted.

Example error in Eventvwr:
Event ID: 1034
The disk associated with cluster disk resource 'Disk Q: L:' could not be found. The expected signature of the disk was 28D73749. If the disk was removed from the server cluster, the resource should be deleted. If the disk was replaced, the resource must be deleted and created again in order to bring the disk online. If the disk has not been removed or replaced, it may be inaccessible at this time because it is reserved by another server cluster node.


Also Attache is a bunch of error information from the cluster error services.

Perhaps someone with a bigger brain then me can decipher?

Thanks,
Post
adc:af8.08/04[09:17:56.375](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:17:56.375](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:af8.08/04[09:17:56.375](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).

adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.

adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.

adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 1296 for 'DwTerminateThread'.

adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.

adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.

adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '7c8edd60-c45c-44de-ac2b-3751677563f9' from resource.

adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property 'e75836bb-6f55-46d3-8060-879bbe618e65' from resource.

adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16

adc:510.08/04[09:17:56.375](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1

adc:510.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EventLogging: First Storage Group/Mailbox Database (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed.  Error Code: 1.

adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.

adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.

adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.

adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).

adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.

adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.

adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 612 for 'DwTerminateThread'.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '7c8edd60-c45c-44de-ac2b-3751677563f9' from resource.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property '7afce291-6cb7-4648-aa85-400eeaffb36b' from resource.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16

adc:264.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1

adc:264.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EventLogging: First Storage Group/bes (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed.  Error Code: 1.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.

adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.

adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).

adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.

adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.

adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 1368 for 'DwTerminateThread'.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '8a1a2af4-860f-4510-8ff7-2cd071b8b045' from resource.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property '46243d7f-fd85-4aea-b787-71f90ae585b7' from resource.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16

adc:558.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1

adc:558.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EventLogging: ExchangeBack/Retired Users (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed.  Error Code: 1.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.

adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.

adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).

adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:17:56.390](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).

adc:578.08/04[09:18:08.609](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.

974:a54.08/04[09:18:08.609](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4

974:a54.08/04[09:18:08.609](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item

adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:08.609](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:aec.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:08.609](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:d08.08/04[09:18:20.781](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.

974:a54.08/04[09:18:20.781](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4

974:a54.08/04[09:18:20.781](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item

adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:20.781](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:af4.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af4.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af4.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af4.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:20.781](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:d28.08/04[09:18:32.937](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.

974:a54.08/04[09:18:32.937](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4

974:a54.08/04[09:18:32.937](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item

adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:32.937](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:18:32.937](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

974:a54.08/04[09:18:32.937](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.

adc:b4c.08/04[09:18:32.937](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.

adc:f44.08/04[09:19:01.781](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.

974:a54.08/04[09:19:01.781](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4

974:a54.08/04[09:19:01.781](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item

adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:01.781](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:01.781](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

974:a54.08/04[09:19:01.781](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.

adc:f34.08/04[09:19:01.781](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.

adc:f88.08/04[09:19:30.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.

974:a54.08/04[09:19:30.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4

974:a54.08/04[09:19:30.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item

adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:19:30.796](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:19:30.796](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

974:a54.08/04[09:19:30.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.

adc:f7c.08/04[09:19:30.796](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.

adc:fc4.08/04[09:19:59.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.

974:a54.08/04[09:19:59.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4

974:a54.08/04[09:19:59.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item

adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:59.796](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:19:59.796](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

974:a54.08/04[09:19:59.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.

adc:fb8.08/04[09:19:59.796](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.

adc:3d4.08/04[09:20:28.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.

974:a54.08/04[09:20:28.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4

974:a54.08/04[09:20:28.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item

adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)

adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:20:28.796](000000) ERR  Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:aec.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.

adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.

adc:af8.08/04[09:20:28.796](000000) ERR  Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.

974:a54.08/04[09:20:28.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.

adc:0c4.08/04[09:20:28.796](000000) ERR  IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.

974:980.08/12[09:02:31.571](000000) WARN [INIT] The cluster service is shutting down.

Open in new window

0
Comment
Question by:Posthumous
  • 3
  • 3
7 Comments
 
LVL 14

Expert Comment

by:Shabarinath Ramadasan
ID: 33445455
From the node which is having this issue, Can you check if all the disks mentioned are available on the Disk Manager? If you have multipath software (HDLM or Powerpath), Check the status on that too.

Go to Disk Manager and click on Rescan- This will do a scan and all the connected disks will be showed.
Make a comparison using the Size and LUN between two nodes.

Also, I would suggest you to try bringing everything offline -> Failover -> take the disk resource online one by one and see how it goes.

Good luck.
Shaba
0
 
LVL 4

Expert Comment

by:sivanov
ID: 33445497
hello ,
you forget to give as the OS of the servers :-), but since the area is win2003 i suppose they are win2k3:
i do not thing that you need someone with more brain, just more wrinkled.
What about this KB
http://support.microsoft.com/kb/280425 - an old but pretty usefull one - did you check it?
or this three - http://support.microsoft.com/kb/243195
and
http://support.microsoft.com/kb/886800.
and http://support.microsoft.com/kb/293778
0
 
LVL 14

Expert Comment

by:Shabarinath Ramadasan
ID: 33445502
I would also suggest you to bring both servers down, wait for a min and then start just the problamatic node. For saferside, Dismount all DBs and check for do not mount at startup before shutting down.
Then see how the cluster behaves.

Good luck
Shaba
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 1

Author Comment

by:Posthumous
ID: 33445511

Good Morning!

When I try to fail over the cluster to node2, the cluster mgr seems to time out completely on that unit.
On Node1 (working node) I can see the listed activity above happening on the node2 but eventually the 3 items listed show as failed.  I believe (and I can't verify this until later after working hours) that the Drive letters are still visable on node1 but not accessable on node1 while this is all going on.  Eventually I have to move all the items back over to node1 to get access to the drives again and bring the exchange folders back online.

From what I can see, the disks never become available on Node2 for the exchange folders to be accessed (which is I believe the root of the issue).
0
 
LVL 1

Author Comment

by:Posthumous
ID: 33445523
Yes this is a server2003 cluster.
Sorry should have mentioned it.
0
 
LVL 14

Expert Comment

by:Shabarinath Ramadasan
ID: 33447783
Thanks for describing more on this.
Once the failover happens from Node 2 to Node1, Node2 should release the disk ownership and Node1 should get it. If this process failes, Node1 wont make the disk resource online.

The best option is to shutdown both nodes. Wait for a min. Start the problematic node and see if it helps.
Also, make sure you dismount the DBs and check "Dont mount this DB at startup" to safegaurd the DB.

Good Luck
Shaba
0
 
LVL 1

Accepted Solution

by:
Posthumous earned 0 total points
ID: 33501868
Problem turned out to be a bad Iscsi interface driver on the cluster, the driver wasn't allowing the storage device to populate properly onto the server.  The drivers were showing as connected in the device manager however no access was going through.

Removing the drivers and reinstalling the latest version from the vendor along with 2 software patches to the management software, 3 reboots and a phone call to the vendor managed to sort it out.

Thanks for the advice all.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Following basic email etiquette rules will help you write a professional email and achieve a good, lasting impression with your contacts.
Learn to move / copy / export exchange contacts to iPhone without using any software. Also see the issues in configuration of exchange with iPhone to migrate contacts.
To add imagery to an HTML email signature, you have two options available to you. You can either add a logo/image by embedding it directly into the signature or hosting it externally and linking to it. The vast majority of email clients display l…
This video discusses moving either the default database or any database to a new volume.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now