cdeblois
asked on
Exchange 2013 DAG Automatic failover
I have two Exchange 2013 Servers in the same site and one Witness server also in the same site. I have configured the DAG and replication is working. When I manage the Cluster from the msc and simulate a failure I get an error message and automatic failover does not occur. I believe I am missing something in my configuration but can't seems to pinpoint it. The error message I receive is "Cluster resource 'Cluster Name' of type 'Network Name' in clustered role 'Cluster Group' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet."
Event ID 1069
Both Exchange servers appear online. Both are running Exchange 2013 on Windows 2012 Standard. The Witness server is running Windows 2008 R2 enterprise
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet."
Event ID 1069
Both Exchange servers appear online. Both are running Exchange 2013 on Windows 2012 Standard. The Witness server is running Windows 2008 R2 enterprise
ASKER
Hi Will Thank you so much for responding. In answer to your questions I can ping the DAG VIP by both IP and by name and it responds. the replication network is up. When you say failover clustering needs to have all services up, What services are you referring to. When I set up the DAG I did no configuration on the Failover Cluster management console it was all there. Is there something I should be adding?
Emily
Emily
You are correct, you shouldn't have had to do anything. I was just saying to check the Failover clustering and make sure that all of the Exchange services are UP based on the cluster.
Will.
Will.
ASKER
Yes services are up and running. When I remote to the DAG I get the active server as well. I have installed and configured clustering with Windows 2008 and Exchange 2007 and it's so very different. What I see on the Cluster console is quite different as well. I am including a picture of the Current console for this network for your review. To me it seems like I'm missing something. Hopefully this will help some. My concern is that although the DB's are replicating with no problem when simulating a failure on the Cluster console I get the following error and mail stops flowing:
" Cluster resource 'Cluster Name' of type 'Network Name' in clustered role 'Cluster Group' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet."
When I run Get-ClusterResource on PS I get the following error:
" Get-ClusterResource : You do not have administrative privileges on the cluster. Contact your network admini
request access.
Access is denied
At line:1 char:1
+ Get-ClusterResource
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : AuthenticationError: (:) [Get-ClusterResource], ClusterCmdletException
+ FullyQualifiedErrorId : ClusterAccessDenied,Micros oft.Failov erClusters .PowerShel l.GetResou rceCommand "
" Cluster resource 'Cluster Name' of type 'Network Name' in clustered role 'Cluster Group' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet."
When I run Get-ClusterResource on PS I get the following error:
" Get-ClusterResource : You do not have administrative privileges on the cluster. Contact your network admini
request access.
Access is denied
At line:1 char:1
+ Get-ClusterResource
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : AuthenticationError: (:) [Get-ClusterResource], ClusterCmdletException
+ FullyQualifiedErrorId : ClusterAccessDenied,Micros
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Update on this question:
Ok I fixed the permission issue. This is what the current scenario is. 2 Exchange Servers. Exchange 1 has all active copies of DB's except 1, Exchange 2 has 1 active copy. When I stop information store on Exchange 1, none of the DB's mount on exchange 2, if I stop the information store on exchange 2 the DB mounts on exchange one. Get-DatabaseAvailabilityGr oup | fl provides the following:
RunspaceId : 1b3139f2-e133-4a47-8695-b8 a71886bbf8
Name : ExchDAG
Servers : {CHVMEXCHANGE2, CHVMEXCHANGE1}
WitnessServer : sapserver.xxx.local
WitnessDirectory : C:\DAGFileShareWitnesses\E xchDAG.xxx .local
AlternateWitnessServer :
AlternateWitnessDirectory :
NetworkCompression : InterSubnetOnly
NetworkEncryption : InterSubnetOnly
ManualDagNetworkConfigurat ion : False
DatacenterActivationMode : DagOnly
StoppedMailboxServers : {}
StartedMailboxServers : {CHVMExchange1.xxx.local, CHVMExchange2.xxx.local}
DatabaseAvailabilityGroupI pv4Address es : {192.168.112.86}
DatabaseAvailabilityGroupI pAddresses : {192.168.112.86}
AllowCrossSiteRpcClientAcc ess : False
OperationalServers :
PrimaryActiveManager :
ServersInMaintenance :
ServersInDeferredRecovery :
ThirdPartyReplication : Disabled
ReplicationPort : 64327
NetworkNames : {}
WitnessShareInUse :
DatabaseAvailabilityGroupC onfigurati on :
AutoDagSchemaVersion : 1.0
AutoDagDatabaseCopiesPerDa tabase : 1
AutoDagDatabaseCopiesPerVo lume : 1
AutoDagTotalNumberOfDataba ses : 0
AutoDagTotalNumberOfServer s : 0
AutoDagDatabasesRootFolder Path : C:\ExchangeDatabases
AutoDagVolumesRootFolderPa th : C:\ExchangeVolumes
AutoDagAllServersInstalled : False
AutoDagAutoReseedEnabled : True
AutoDagDiskReclaimerEnable d : True
ReplayLagManagerEnabled : False
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=ExchDAG,CN=Database Availability Groups,CN=Exchange Administrative Group
(FYDIBOHF23SPDLT),CN=Admin istrative Groups,CN=xxx,CN=Microsoft
Exchange,CN=Services,CN=Co nfiguratio n,DC=xxx,D C=local
Identity : ExchDAG
Guid : b67e4a1b-eee7-40c4-bf08-37 a17d04c8e1
ObjectCategory : xxx.local/Configuration/Sc hema/ms-Ex ch-MDB-Ava ilability- Group
ObjectClass : {top, msExchMDBAvailabilityGroup }
WhenChanged : 1/12/2014 8:16:06 PM
WhenCreated : 1/2/2014 12:53:21 PM
WhenChangedUTC : 1/13/2014 1:16:06 AM
WhenCreatedUTC : 1/2/2014 5:53:21 PM
OrganizationId :
OriginatingServer : CHVMADS1.xxx.local
IsValid : True
ObjectState : Unchanged
Ok I fixed the permission issue. This is what the current scenario is. 2 Exchange Servers. Exchange 1 has all active copies of DB's except 1, Exchange 2 has 1 active copy. When I stop information store on Exchange 1, none of the DB's mount on exchange 2, if I stop the information store on exchange 2 the DB mounts on exchange one. Get-DatabaseAvailabilityGr
RunspaceId : 1b3139f2-e133-4a47-8695-b8
Name : ExchDAG
Servers : {CHVMEXCHANGE2, CHVMEXCHANGE1}
WitnessServer : sapserver.xxx.local
WitnessDirectory : C:\DAGFileShareWitnesses\E
AlternateWitnessServer :
AlternateWitnessDirectory :
NetworkCompression : InterSubnetOnly
NetworkEncryption : InterSubnetOnly
ManualDagNetworkConfigurat
DatacenterActivationMode : DagOnly
StoppedMailboxServers : {}
StartedMailboxServers : {CHVMExchange1.xxx.local, CHVMExchange2.xxx.local}
DatabaseAvailabilityGroupI
DatabaseAvailabilityGroupI
AllowCrossSiteRpcClientAcc
OperationalServers :
PrimaryActiveManager :
ServersInMaintenance :
ServersInDeferredRecovery :
ThirdPartyReplication : Disabled
ReplicationPort : 64327
NetworkNames : {}
WitnessShareInUse :
DatabaseAvailabilityGroupC
AutoDagSchemaVersion : 1.0
AutoDagDatabaseCopiesPerDa
AutoDagDatabaseCopiesPerVo
AutoDagTotalNumberOfDataba
AutoDagTotalNumberOfServer
AutoDagDatabasesRootFolder
AutoDagVolumesRootFolderPa
AutoDagAllServersInstalled
AutoDagAutoReseedEnabled : True
AutoDagDiskReclaimerEnable
ReplayLagManagerEnabled : False
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=ExchDAG,CN=Database Availability Groups,CN=Exchange Administrative Group
(FYDIBOHF23SPDLT),CN=Admin
Exchange,CN=Services,CN=Co
Identity : ExchDAG
Guid : b67e4a1b-eee7-40c4-bf08-37
ObjectCategory : xxx.local/Configuration/Sc
ObjectClass : {top, msExchMDBAvailabilityGroup
WhenChanged : 1/12/2014 8:16:06 PM
WhenCreated : 1/2/2014 12:53:21 PM
WhenChangedUTC : 1/13/2014 1:16:06 AM
WhenCreatedUTC : 1/2/2014 5:53:21 PM
OrganizationId :
OriginatingServer : CHVMADS1.xxx.local
IsValid : True
ObjectState : Unchanged
The databases that reside on Exchange 1 do you have them located on separate drives (i.e. logs = F:\ data = G:)? If this is true you need to setup the same directory strucutre on the Exchange 2 server. If you do not have this it will not work.
You will also need to make sure that you have actually set the Exchange 2 server as a copy for the mailboxes on Exchange 1. See below screenshot...
Adding Mailbox Copy to Another server in the DAG.
Will.
You will also need to make sure that you have actually set the Exchange 2 server as a copy for the mailboxes on Exchange 1. See below screenshot...
Adding Mailbox Copy to Another server in the DAG.
Will.
ASKER
Hi Will. Both of these servers are identical. They are vitual machines and the exchange logs and DB are both situated on the C Drive in the default location. The DAG contains both servers and when I ran test-replicationhealth | fl I got the following:
[PS] C:\Windows\system32>test-r eplication health | ft
Server Check Result Error
------ ----- ------ -----
CHVMEXCHANGE1 ClusterService Passed
CHVMEXCHANGE1 ReplayService Passed
CHVMEXCHANGE1 ActiveManager Passed
CHVMEXCHANGE1 TasksRpcListener Passed
CHVMEXCHANGE1 TcpListener Passed
CHVMEXCHANGE1 ServerLocatorService Passed
CHVMEXCHANGE1 DagMembersUp Passed
CHVMEXCHANGE1 ClusterNetwork Passed
CHVMEXCHANGE1 QuorumGroup Passed
CHVMEXCHANGE1 FileShareQuorum Passed
CHVMEXCHANGE1 DatabaseRedundancy Passed
CHVMEXCHANGE1 DatabaseAvailability Passed
CHVMEXCHANGE1 DBCopySuspended Passed
CHVMEXCHANGE1 DBCopyFailed Passed
CHVMEXCHANGE1 DBInitializing Passed
CHVMEXCHANGE1 DBDisconnected Passed
CHVMEXCHANGE1 DBLogCopyKeepingUp Passed
CHVMEXCHANGE1 DBLogReplayKeepingUp Passed
[PS] C:\Windows\system32>test-r
Server Check Result Error
------ ----- ------ -----
CHVMEXCHANGE1 ClusterService Passed
CHVMEXCHANGE1 ReplayService Passed
CHVMEXCHANGE1 ActiveManager Passed
CHVMEXCHANGE1 TasksRpcListener Passed
CHVMEXCHANGE1 TcpListener Passed
CHVMEXCHANGE1 ServerLocatorService Passed
CHVMEXCHANGE1 DagMembersUp Passed
CHVMEXCHANGE1 ClusterNetwork Passed
CHVMEXCHANGE1 QuorumGroup Passed
CHVMEXCHANGE1 FileShareQuorum Passed
CHVMEXCHANGE1 DatabaseRedundancy Passed
CHVMEXCHANGE1 DatabaseAvailability Passed
CHVMEXCHANGE1 DBCopySuspended Passed
CHVMEXCHANGE1 DBCopyFailed Passed
CHVMEXCHANGE1 DBInitializing Passed
CHVMEXCHANGE1 DBDisconnected Passed
CHVMEXCHANGE1 DBLogCopyKeepingUp Passed
CHVMEXCHANGE1 DBLogReplayKeepingUp Passed
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
On Get-DatabaseAvailabilityGr oup -Status |fl I get:
RunspaceId : 213d11e1-9cb8-480c-baab-b0 93673ce055
Name : ExchDAG
Servers : {CHVMEXCHANGE2, CHVMEXCHANGE1}
WitnessServer : sapserver.commlogik.local
WitnessDirectory : C:\DAGFileShareWitnesses\E xchDAG.com mlogik.loc al
AlternateWitnessServer :
AlternateWitnessDirectory :
NetworkCompression : InterSubnetOnly
NetworkEncryption : InterSubnetOnly
ManualDagNetworkConfigurat ion : False
DatacenterActivationMode : DagOnly
StoppedMailboxServers : {}
StartedMailboxServers : {CHVMExchange1.commlogik.l ocal, CHVMExchange2.commlogik.lo cal}
DatabaseAvailabilityGroupI pv4Address es : {192.168.112.86}
DatabaseAvailabilityGroupI pAddresses : {192.168.112.86}
AllowCrossSiteRpcClientAcc ess : False
OperationalServers : {CHVMEXCHANGE1, CHVMEXCHANGE2}
PrimaryActiveManager : CHVMEXCHANGE2
ServersInMaintenance : {}
ServersInDeferredRecovery : {}
ThirdPartyReplication : Disabled
ReplicationPort : 64327
NetworkNames : {MapiDagNetwork, ReplicationDagNetwork01}
WitnessShareInUse : Primary
DatabaseAvailabilityGroupC onfigurati on :
AutoDagSchemaVersion : 1.0
AutoDagDatabaseCopiesPerDa tabase : 1
AutoDagDatabaseCopiesPerVo lume : 1
AutoDagTotalNumberOfDataba ses : 0
AutoDagTotalNumberOfServer s : 0
AutoDagDatabasesRootFolder Path : C:\ExchangeDatabases
AutoDagVolumesRootFolderPa th : C:\ExchangeVolumes
AutoDagAllServersInstalled : False
AutoDagAutoReseedEnabled : True
AutoDagDiskReclaimerEnable d : True
ReplayLagManagerEnabled : False
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=ExchDAG,CN=Database Availability Groups,CN=Exchange Administrative Group
(FYDIBOHF23SPDLT),CN=Admin istrative Groups,CN=Commlogik,CN=Mic rosoft
Exchange,CN=Services,CN=Co nfiguratio n,DC=comml ogik,DC=lo cal
Identity : ExchDAG
Guid : b67e4a1b-eee7-40c4-bf08-37 a17d04c8e1
ObjectCategory : commlogik.local/Configurat ion/Schema /ms-Exch-M DB-Availab ility-Grou p
ObjectClass : {top, msExchMDBAvailabilityGroup }
WhenChanged : 1/12/2014 8:16:28 PM
WhenCreated : 1/2/2014 12:53:21 PM
WhenChangedUTC : 1/13/2014 1:16:28 AM
WhenCreatedUTC : 1/2/2014 5:53:21 PM
OrganizationId :
OriginatingServer : HERA.commlogik.local
IsValid : True
ObjectState : Unchanged
On Get-mailboxdatabase I get:
Name Server Recovery ReplicationType
---- ------ -------- ---------------
MDB-2013_Corp CHVMEXCHANGE1 False Remote
MDB-2013_VIP CHVMEXCHANGE1 False Remote
MDB-2013_Col CHVMEXCHANGE1 False Remote
MDB-2013_Mex CHVMEXCHANGE1 False Remote
MDB-2013_Ecu CHVMEXCHANGE1 False Remote
MDB_2013_Arg CHVMEXCHANGE1 False Remote
Mailbox Database 0651537422 CHVMEXCHANGE2 False Remote
All servers appear in the first. I can't test the move at this moment as these are production servers and I don't want to stop users from working.
RunspaceId : 213d11e1-9cb8-480c-baab-b0
Name : ExchDAG
Servers : {CHVMEXCHANGE2, CHVMEXCHANGE1}
WitnessServer : sapserver.commlogik.local
WitnessDirectory : C:\DAGFileShareWitnesses\E
AlternateWitnessServer :
AlternateWitnessDirectory :
NetworkCompression : InterSubnetOnly
NetworkEncryption : InterSubnetOnly
ManualDagNetworkConfigurat
DatacenterActivationMode : DagOnly
StoppedMailboxServers : {}
StartedMailboxServers : {CHVMExchange1.commlogik.l
DatabaseAvailabilityGroupI
DatabaseAvailabilityGroupI
AllowCrossSiteRpcClientAcc
OperationalServers : {CHVMEXCHANGE1, CHVMEXCHANGE2}
PrimaryActiveManager : CHVMEXCHANGE2
ServersInMaintenance : {}
ServersInDeferredRecovery : {}
ThirdPartyReplication : Disabled
ReplicationPort : 64327
NetworkNames : {MapiDagNetwork, ReplicationDagNetwork01}
WitnessShareInUse : Primary
DatabaseAvailabilityGroupC
AutoDagSchemaVersion : 1.0
AutoDagDatabaseCopiesPerDa
AutoDagDatabaseCopiesPerVo
AutoDagTotalNumberOfDataba
AutoDagTotalNumberOfServer
AutoDagDatabasesRootFolder
AutoDagVolumesRootFolderPa
AutoDagAllServersInstalled
AutoDagAutoReseedEnabled : True
AutoDagDiskReclaimerEnable
ReplayLagManagerEnabled : False
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=ExchDAG,CN=Database Availability Groups,CN=Exchange Administrative Group
(FYDIBOHF23SPDLT),CN=Admin
Exchange,CN=Services,CN=Co
Identity : ExchDAG
Guid : b67e4a1b-eee7-40c4-bf08-37
ObjectCategory : commlogik.local/Configurat
ObjectClass : {top, msExchMDBAvailabilityGroup
WhenChanged : 1/12/2014 8:16:28 PM
WhenCreated : 1/2/2014 12:53:21 PM
WhenChangedUTC : 1/13/2014 1:16:28 AM
WhenCreatedUTC : 1/2/2014 5:53:21 PM
OrganizationId :
OriginatingServer : HERA.commlogik.local
IsValid : True
ObjectState : Unchanged
On Get-mailboxdatabase I get:
Name Server Recovery ReplicationType
---- ------ -------- ---------------
MDB-2013_Corp CHVMEXCHANGE1 False Remote
MDB-2013_VIP CHVMEXCHANGE1 False Remote
MDB-2013_Col CHVMEXCHANGE1 False Remote
MDB-2013_Mex CHVMEXCHANGE1 False Remote
MDB-2013_Ecu CHVMEXCHANGE1 False Remote
MDB_2013_Arg CHVMEXCHANGE1 False Remote
Mailbox Database 0651537422 CHVMEXCHANGE2 False Remote
All servers appear in the first. I can't test the move at this moment as these are production servers and I don't want to stop users from working.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thank you Will I will keep you informed. Thank you!!
ASKER
Hi Will sorry for the delay in responding. I wound up calling Microsoft on this issue and after several ours of troubleshooting they found that the network had a DC that was improperly removed and it was causing the replication errors and the failover errors as a result. Once the Dc was removed properly, I updated Exchange 2 to CU2 so that both servers were running the same version. It was at this point that failover was successful. Thank you so much for your help on this issue :)
Perfect, gald to have helped. In the mean time you can use the above commands to perform all of your testing/verification of services.
Will.
Will.
ASKER
LOl Trust me I have an excel sheet loaded with all those goodies and some I got from the microsoft call as well :) Again thank you!
If you run the command
Move-activemailboxdatabase
Does the command complete successfully?
Also run the command...
Get-databaseavailibilitygr
Check the active server for the database and also does it show both servers under operational servers?
Things to consider for DAG
- DAG VIP needs to be pingable
- DAG replication network needs to be up
- Failover clustering needs to have all services "up"
Will.