Solved

Issues joining a new Mailbox/Hub server to the Database Availability Group

Posted on 2013-06-13
4
1,166 Views
Last Modified: 2013-07-22
I have two servers currently in the DAG and I'm trying to add a 3rd.   I'm getting the error  below when trying to add that 3rd server. I'm seeing answers of all kinds on Forums, I've tried changing some NIC properties, I do have the same OS on all 3 servers (Win 2008 Ent.), I can't think of other things I've tried at this moment..   Hopefully someone else has run into this issue and fixed it.  Here is the error:

Summary: 1 item(s). 0 succeeded, 1 failed.
Elapsed time: 00:06:23


SEMEX06
Failed

Error:
A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed. [Server: SEMEX05.sug.com]

An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed..

This operation returned because the timeout period expired
Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.140).aspx?v=14.2.342.0&t=exchgf1&e=ms.exch.err.ExC9C315

Warning:
The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-06-13_13-23-54.076_add-databaseavailabiltygroupserver.log".


Exchange Management Shell command attempted:
Add-DatabaseAvailabilityGroupServer -MailboxServer 'SEMEX06' -Identity 'ExchDag'

Elapsed Time: 00:06:23
0
Comment
Question by:jonasdurst
  • 3
4 Comments
 

Author Comment

by:jonasdurst
ID: 39244695
log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-06-13_13-23-54.076_add-databaseavailabiltygroupserver.log"
----------------------------------------------------------------------------------------------------------------------------------------

add-databaseavailabiltygroupserver started on machine SEMCAS01.
[2013-06-13T13:23:54] add-dagserver started
[2013-06-13T13:23:54] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2013-06-13T13:23:54] Option 'Identity' = 'ExchDag'.
[2013-06-13T13:23:54] Option 'MailboxServer' = 'SEMEX06'.
[2013-06-13T13:23:54] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2013-06-13T13:23:54] Option 'WhatIf' = ''.
[2013-06-13T13:23:54] Process: w3wp w3wp.exe:3248.
[2013-06-13T13:23:54] User context = 'NT AUTHORITY\SYSTEM'.
[2013-06-13T13:23:54]   Member of group 'Everyone'.
[2013-06-13T13:23:54]   Member of group 'BUILTIN\Users'.
[2013-06-13T13:23:54]   Member of group 'NT AUTHORITY\SERVICE'.
[2013-06-13T13:23:54]   Member of group 'CONSOLE LOGON'.
[2013-06-13T13:23:54]   Member of group 'NT AUTHORITY\Authenticated Users'.
[2013-06-13T13:23:54]   Member of group 'NT AUTHORITY\This Organization'.
[2013-06-13T13:23:54]   Member of group 'BUILTIN\IIS_IUSRS'.
[2013-06-13T13:23:54]   Member of group 'LOCAL'.
[2013-06-13T13:23:54]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
[2013-06-13T13:23:54]   Member of group 'BUILTIN\Administrators'.
[2013-06-13T13:23:54] Updated Progress 'Validating the parameters.' 2%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] Mailbox server: value passed in = SEMEX06, mailboxServer.Name = SEMEX06, mailboxServer.Fqdn = SEMEX06.sug.com
[2013-06-13T13:23:54] LogClussvcState: clussvc is Stopped on SEMEX06.sug.com. Exception (if any) = none
[2013-06-13T13:23:54] The IP addresses for the DAG are (blank means DHCP): 10.1.5.162
[2013-06-13T13:23:54] Looking up IP addresses for ExchDag.
[2013-06-13T13:23:54]   ExchDag = [ 10.1.5.162 ].
[2013-06-13T13:23:54] Looking up IP addresses for semex06.
[2013-06-13T13:23:54]   semex06 = [ 10.1.5.160 ].
[2013-06-13T13:23:54] Looking up IP addresses for SEMEX06.sug.com.
[2013-06-13T13:23:54]   SEMEX06.sug.com = [ 10.1.5.160 ].
[2013-06-13T13:23:54] DAG ExchDag has 2 servers:
[2013-06-13T13:23:54] DAG ExchDag contains server SEMEX05.
[2013-06-13T13:23:54] DAG ExchDag contains server SEMEX04.
[2013-06-13T13:23:54] Updated Progress 'Checking if Mailbox server 'SEMEX06' is in a database availability group.' 4%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(SEMEX06.sug.com) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed... This is OK.
[2013-06-13T13:23:54] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2013-06-13T13:23:54] DumpClusterTopology: Opening remote cluster ExchDag.
[2013-06-13T13:23:54] Dumping the cluster by connecting to: ExchDag.
[2013-06-13T13:23:54] The cluster's name is: ExchDag.
[2013-06-13T13:23:54] Groups
[2013-06-13T13:23:54]     group: Available Storage [not a CMS]
[2013-06-13T13:23:54]         OwnerNode: SEMEX04.sug.com
[2013-06-13T13:23:54]         State: Offline
[2013-06-13T13:23:54]     group: Cluster Group [Cluster Main Group]
[2013-06-13T13:23:54]         OwnerNode: SEMEX05.sug.com
[2013-06-13T13:23:54]         State: Online
[2013-06-13T13:23:54]             Resource: IPv4 Static Address 1 (Cluster Group) [Online, type = IP Address, PossibleOwners = SEMEX04,SEMEX05 ]
[2013-06-13T13:23:54]                 Address = [10.1.5.162]
[2013-06-13T13:23:54]                     EnableDhcp = [0]
[2013-06-13T13:23:54]                     Network = [Cluster Network 1]
[2013-06-13T13:23:54]             Resource: Cluster Name [Online, type = Network Name, PossibleOwners = SEMEX04,SEMEX05 ]
[2013-06-13T13:23:54]                 NetName = [EXCHDAG]
[2013-06-13T13:23:54]                     RequireKerberos = 1. Computer account created on DC ''.
[2013-06-13T13:23:54]             Resource: File Share Witness (\\semfs01.sug.com\ExchDag.sug.com) [Online, type = File Share Witness, PossibleOwners = SEMEX04,SEMEX05 ]
[2013-06-13T13:23:54] Nodes
[2013-06-13T13:23:54]     node: SEMEX04.sug.com [ state = Up ]
[2013-06-13T13:23:54]     node: SEMEX05.sug.com [ state = Up ]
[2013-06-13T13:23:54] Subnets
[2013-06-13T13:23:54]     Name(Cluster Network 1), Mask(10.1.0.0/16), Role(ClusterNetworkRoleInternalAndClient)
[2013-06-13T13:23:54]         NIC 10.1.5.159 on Node SEMEX05 in State=Up
[2013-06-13T13:23:54]         NIC 10.1.5.158 on Node SEMEX04 in State=Up
[2013-06-13T13:23:54]     Name(Cluster Network 3), Mask(10.150.0.0/16), Role(ClusterNetworkRoleInternalAndClient)
[2013-06-13T13:23:54]         NIC 10.150.0.120 on Node SEMEX04 in State=Up
[2013-06-13T13:23:54]         NIC 10.150.0.122 on Node SEMEX05 in State=Up
[2013-06-13T13:23:54] Opening the cluster on nodes [semex05, semex04].
[2013-06-13T13:23:54] Other mailbox servers in the DAG are already members of cluster 'ExchDag'
[2013-06-13T13:23:54] The server SEMEX06 does not belong to a cluster, and the other servers belong to ExchDag.
[2013-06-13T13:23:54] Successfully resolved the servers based on the stopped servers list.
[2013-06-13T13:23:54] The following servers are in the StartedServers list (The list is the StartedServers property of the DAG in AD):
[2013-06-13T13:23:54] The following servers are in the StoppedServers list:
[2013-06-13T13:23:54] Verifiying that the members of database availability group 'ExchDag' are also members of the cluster.
[2013-06-13T13:23:54] Verifying that the members of cluster 'ExchDag' are also members of the database availability group.
[2013-06-13T13:23:54] According to GetNodeClusterState(), the server SEMEX06 is NotConfigured.
[2013-06-13T13:23:54] The CNO is currently Online.
[2013-06-13T13:23:54] InternalValidate() done.
[2013-06-13T13:23:54] Updated Progress 'Adding server 'SEMEX06' to database availability group 'ExchDag'.' 6%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] Updated Progress 'Adding server 'SEMEX06' to the cluster.' 8%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:30:17] The following log entry comes from a different process that's running on machine 'SEMEX05.sug.com'. BEGIN
[2013-06-13T13:30:17] [2013-06-13T13:23:54] Opening a local AmCluster handle.
[2013-06-13T13:23:54] Updated Progress 'Adding server 'semex06' to database availability group 'ExchDag'.' 2%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:54] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:54] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = ExchDag, dwStatus = 0x0 )
[2013-06-13T13:23:55] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:55] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = SEMEX06.sug.com, dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:26:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 100, szObjectName = SEMEX06, dwStatus = 0x5b4 )
[2013-06-13T13:26:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:30:17] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = , dwStatus = 0x0 )

[2013-06-13T13:30:17] The preceding log entry comes from a different process running on computer 'SEMEX05.sug.com'. END
[2013-06-13T13:30:17] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-06-13_13-23-54.076_add-databaseavailabiltygroupserver.log".
[2013-06-13T13:30:17] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed.. ---> System.ComponentModel.Win32Exception: This operation returned because the timeout period expired
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
   --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) ---
   at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
   at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
   --- End of stack trace on server (SEMEX05.sug.com) ---
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(AmServerName serverName, String databaseName, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(AmServerName serverName, Nullable`1 dbGuid, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
[2013-06-13T13:30:17] Updated Progress 'Done!' 100%.
[2013-06-13T13:30:17] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().
0
 
LVL 4

Assisted Solution

by:Alexander Kireev
Alexander Kireev earned 500 total points
ID: 39258678
Hello,

Do your DAG members use two network adapters (first for DAG, second for data)?
Do you sure that you haven't issues with ip-addresses? Could you send a screenshot of your DAG networks (Organization... - Mailbox - DAG - Networks) and ipconfig of new server?

Are SEMEX06 and other DAG members has same Mailbox/HUB role or different roles?
Is SEMEX06 has the same Exchange version (include RU)?

Send me a result of cmdlt "Get-DatabaseAvailabilityGroup | fl".

http://exchangeserverpro.com/exchange-server-2010-database-availability-group-installation-step-by-step/
0
 

Accepted Solution

by:
jonasdurst earned 0 total points
ID: 39327976
Since we are going with an entirely new domain I'm rebuilding the exchange environment from the ground up, no longer have a need to add the additional MDB.   I have physical servers, going to build two CAS/HUB servers and then two MDB servers.
0
 

Author Closing Comment

by:jonasdurst
ID: 39345319
Since I've avoided the issue buy going down another road I technically didn't solve the issue
0

Featured Post

Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Lotus Notes – formerly IBM Notes – is an email client application, while IBM Domino (earlier Lotus Domino) is an email server. The client possesses a set of features that are even more advanced as compared to that of Outlook. Likewise, IBM Domino is…
Follow this checklist to learn more about the 15 things you should never include in an email signature from personal quotes, animated gifs and out-of-date marketing content.
To show how to create a transport rule in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Rules tab.:  To cr…
The viewer will learn how to create two correlated normally distributed random variables in Excel, use a normal distribution to simulate the return on different levels of investment in each of the two funds over a period of ten years, and, create a …

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question