Cluster Services not Mounting HDD's properly?
Good Morning folks, hope someone has an idea on this one because i'm stumped!
We've got a cluster running for our exchange servers, and at the moment it's running fine with node1 handling all the work load. However when we try to test the fail over of the databases to node2 we run into a bit of an issue.
Once we take node1 offline, node2 cluster services SEEM to start spinning up.
It's broken intto two parts.
First part includes the following:
Cluster IP
Cluster Name
Exchange HDD mapings (E:,Q:,L:,S)
MSDTC
Part two however is where we seem to be running into issues (tho I suspect an earlier probelm in the sequence).
Part 2 as follows:
Exchange Information Store Instance
Exchange system Attendant Instance
Exchange Storage Group - Retired
Exchange Storage Group - Bes
Exhange Storage Group - Primary Storage Database
IP address
Network Name
The 3 bolded items are the ones that show up as failed in the management window.
However when I check the Win Explorer, I don't actually see the drive mappings from Phase 1 being loaded, ie there's no E/Q/L/S drives being displayed, which obviously would kill the concept of the physcal disks being mounted.
Example error in Eventvwr:
Event ID: 1034
The disk associated with cluster disk resource 'Disk Q: L:' could not be found. The expected signature of the disk was 28D73749. If the disk was removed from the server cluster, the resource should be deleted. If the disk was replaced, the resource must be deleted and created again in order to bring the disk online. If the disk has not been removed or replaced, it may be inaccessible at this time because it is reserved by another server cluster node.
Also Attache is a bunch of error information from the cluster error services.
Perhaps someone with a bigger brain then me can decipher?
Thanks,
Post
adc:af8.08/04[09:17:56.375](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:17:56.375](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:17:56.375](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.
adc:af4.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 1296 for 'DwTerminateThread'.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '7c8edd60-c45c-44de-ac2b-3751677563f9' from resource.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property 'e75836bb-6f55-46d3-8060-879bbe618e65' from resource.
adc:510.08/04[09:17:56.375](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16
adc:510.08/04[09:17:56.375](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1
adc:510.08/04[09:17:56.390](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EventLogging: First Storage Group/Mailbox Database (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed. Error Code: 1.
adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.
adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.
adc:510.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 612 for 'DwTerminateThread'.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '7c8edd60-c45c-44de-ac2b-3751677563f9' from resource.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property '7afce291-6cb7-4648-aa85-400eeaffb36b' from resource.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16
adc:264.08/04[09:17:56.390](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1
adc:264.08/04[09:17:56.390](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EventLogging: First Storage Group/bes (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed. Error Code: 1.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.
adc:264.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=2.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwCreateResourceThread::Increment: g_lThreadCount=1.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwCreateResourceThread: created thread 1368 for 'DwTerminateThread'.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] Thread entered: DwTerminateThread.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ecRPCOffLine: calling offline RPC.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ReadParameters: read StorageGroupGuid property '8a1a2af4-860f-4510-8ff7-2cd071b8b045' from resource.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ReadParameters: read DatabaseGuid property '46243d7f-fd85-4aea-b787-71f90ae585b7' from resource.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] calling EcUnmountDatabase() with flags 16
adc:558.08/04[09:17:56.390](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EcUnmountDatabase() failed and returned 1
adc:558.08/04[09:17:56.390](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EventLogging: ExchangeBack/Retired Users (ECEXCHANGE): The RPC call to the service to take the cluster resource offline failed. Error Code: 1.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] Thread exited: DwTerminateThread. Returning error 1.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] DwResourceThread::decrement:: g_lThreadCount=0.
adc:558.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=1.
adc:af4.08/04[09:17:56.390](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:17:56.390](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:17:56.390](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] State change: from 3 (Offline) to 4 (Failed).
adc:578.08/04[09:18:08.609](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:18:08.609](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:18:08.609](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:08.609](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:08.609](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:08.609](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:18:08.609](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:08.609](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:08.609](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:08.609](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:d08.08/04[09:18:20.781](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:18:20.781](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:18:20.781](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:20.781](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af4.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:18:20.781](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:20.781](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af4.08/04[09:18:20.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af4.08/04[09:18:20.781](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:20.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:20.781](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:d28.08/04[09:18:32.937](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:18:32.937](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:18:32.937](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:32.937](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:32.937](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:32.937](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:18:32.937](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:18:32.937](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:18:32.937](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:18:32.937](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:18:32.937](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:b4c.08/04[09:18:32.937](000000) ERR IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:f44.08/04[09:19:01.781](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:19:01.781](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:19:01.781](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:01.781](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:01.781](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:01.781](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:01.781](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:01.781](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:01.781](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:01.781](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:19:01.781](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:f34.08/04[09:19:01.781](000000) ERR IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:f88.08/04[09:19:30.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:19:30.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:19:30.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:30.796](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:30.796](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:30.796](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:30.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:30.796](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:30.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:30.796](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:19:30.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:f7c.08/04[09:19:30.796](000000) ERR IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:fc4.08/04[09:19:59.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:19:59.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:19:59.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:59.796](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:59.796](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:59.796](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:19:59.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:19:59.796](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:19:59.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:19:59.796](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:19:59.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:fb8.08/04[09:19:59.796](000000) ERR IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
adc:3d4.08/04[09:20:28.796](000000) WARN Network Name <Network Name (ECEXCHANGE)>: Failed to delete server name ECEXCHANGE, status 67.
974:a54.08/04[09:20:28.796](000000) WARN [FM] FmpHandleResourceTransition: Resource Name = 14ac40e6-3b8a-4569-8595-55fc00f1d214 [Network Name (ECEXCHANGE)] old state=129 new state=4
974:a54.08/04[09:20:28.796](000000) WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeResourceControl: Received state change reason 2 (Failover)
adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:20:28.796](000000) ERR Microsoft Exchange System Attendant <Exchange System Attendant Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:20:28.796](000000) ERR Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:20:28.796](000000) ERR Microsoft Exchange Database Instance <First Storage Group/bes (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:aec.08/04[09:20:28.796](000000) WARN Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:aec.08/04[09:20:28.796](000000) ERR Microsoft Exchange Database Instance <ExchangeBack/Retired Users (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Database state (Failed) is failed, skip database unmount rpc.
adc:af8.08/04[09:20:28.796](000000) WARN Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] ExchangeTerminate: Terminate requested.
adc:af8.08/04[09:20:28.796](000000) ERR Microsoft Exchange Information Store <Exchange Information Store Instance (ECEXCHANGE)>: [EXRES] TerminateAndDumpResourceProcess: Process dump and termination skipped as dwServicePid is zero.
974:a54.08/04[09:20:28.796](000000) WARN [FM] Group failure for group <e3d3ccf6-8f11-41a3-a956-3e1e6ee3840c>. Create thread to take offline and move.
adc:0c4.08/04[09:20:28.796](000000) ERR IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
974:980.08/12[09:02:31.571](000000) WARN [INIT] The cluster service is shutting down.
hello ,
you forget to give as the OS of the servers :-), but since the area is win2003 i suppose they are win2k3:
i do not thing that you need someone with more brain, just more wrinkled.
What about this KB
http://support.microsoft.com/kb/280425 - an old but pretty usefull one - did you check it?
or this three - http://support.microsoft.com/kb/243195
and
http://support.microsoft.com/kb/886800.
and http://support.microsoft.com/kb/293778
you forget to give as the OS of the servers :-), but since the area is win2003 i suppose they are win2k3:
i do not thing that you need someone with more brain, just more wrinkled.
What about this KB
http://support.microsoft.com/kb/280425 - an old but pretty usefull one - did you check it?
or this three - http://support.microsoft.com/kb/243195
and
http://support.microsoft.com/kb/886800.
and http://support.microsoft.com/kb/293778
I would also suggest you to bring both servers down, wait for a min and then start just the problamatic node. For saferside, Dismount all DBs and check for do not mount at startup before shutting down.
Then see how the cluster behaves.
Good luck
Shaba
Then see how the cluster behaves.
Good luck
Shaba
ASKER
Good Morning!
When I try to fail over the cluster to node2, the cluster mgr seems to time out completely on that unit.
On Node1 (working node) I can see the listed activity above happening on the node2 but eventually the 3 items listed show as failed. I believe (and I can't verify this until later after working hours) that the Drive letters are still visable on node1 but not accessable on node1 while this is all going on. Eventually I have to move all the items back over to node1 to get access to the drives again and bring the exchange folders back online.
From what I can see, the disks never become available on Node2 for the exchange folders to be accessed (which is I believe the root of the issue).
ASKER
Yes this is a server2003 cluster.
Sorry should have mentioned it.
Sorry should have mentioned it.
Thanks for describing more on this.
Once the failover happens from Node 2 to Node1, Node2 should release the disk ownership and Node1 should get it. If this process failes, Node1 wont make the disk resource online.
The best option is to shutdown both nodes. Wait for a min. Start the problematic node and see if it helps.
Also, make sure you dismount the DBs and check "Dont mount this DB at startup" to safegaurd the DB.
Good Luck
Shaba
Once the failover happens from Node 2 to Node1, Node2 should release the disk ownership and Node1 should get it. If this process failes, Node1 wont make the disk resource online.
The best option is to shutdown both nodes. Wait for a min. Start the problematic node and see if it helps.
Also, make sure you dismount the DBs and check "Dont mount this DB at startup" to safegaurd the DB.
Good Luck
Shaba
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Go to Disk Manager and click on Rescan- This will do a scan and all the connected disks will be showed.
Make a comparison using the Size and LUN between two nodes.
Also, I would suggest you to try bringing everything offline -> Failover -> take the disk resource online one by one and see how it goes.
Good luck.
Shaba