Solved

Issues joining a new Mailbox/Hub server to the Database Availability Group

Posted on 2013-06-13
4
1,186 Views
Last Modified: 2013-07-22
I have two servers currently in the DAG and I'm trying to add a 3rd.   I'm getting the error  below when trying to add that 3rd server. I'm seeing answers of all kinds on Forums, I've tried changing some NIC properties, I do have the same OS on all 3 servers (Win 2008 Ent.), I can't think of other things I've tried at this moment..   Hopefully someone else has run into this issue and fixed it.  Here is the error:

Summary: 1 item(s). 0 succeeded, 1 failed.
Elapsed time: 00:06:23


SEMEX06
Failed

Error:
A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed. [Server: SEMEX05.sug.com]

An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed..

This operation returned because the timeout period expired
Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.140).aspx?v=14.2.342.0&t=exchgf1&e=ms.exch.err.ExC9C315

Warning:
The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-06-13_13-23-54.076_add-databaseavailabiltygroupserver.log".


Exchange Management Shell command attempted:
Add-DatabaseAvailabilityGroupServer -MailboxServer 'SEMEX06' -Identity 'ExchDag'

Elapsed Time: 00:06:23
0
Comment
Question by:jonasdurst
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 

Author Comment

by:jonasdurst
ID: 39244695
log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-06-13_13-23-54.076_add-databaseavailabiltygroupserver.log"
----------------------------------------------------------------------------------------------------------------------------------------

add-databaseavailabiltygroupserver started on machine SEMCAS01.
[2013-06-13T13:23:54] add-dagserver started
[2013-06-13T13:23:54] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2013-06-13T13:23:54] Option 'Identity' = 'ExchDag'.
[2013-06-13T13:23:54] Option 'MailboxServer' = 'SEMEX06'.
[2013-06-13T13:23:54] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2013-06-13T13:23:54] Option 'WhatIf' = ''.
[2013-06-13T13:23:54] Process: w3wp w3wp.exe:3248.
[2013-06-13T13:23:54] User context = 'NT AUTHORITY\SYSTEM'.
[2013-06-13T13:23:54]   Member of group 'Everyone'.
[2013-06-13T13:23:54]   Member of group 'BUILTIN\Users'.
[2013-06-13T13:23:54]   Member of group 'NT AUTHORITY\SERVICE'.
[2013-06-13T13:23:54]   Member of group 'CONSOLE LOGON'.
[2013-06-13T13:23:54]   Member of group 'NT AUTHORITY\Authenticated Users'.
[2013-06-13T13:23:54]   Member of group 'NT AUTHORITY\This Organization'.
[2013-06-13T13:23:54]   Member of group 'BUILTIN\IIS_IUSRS'.
[2013-06-13T13:23:54]   Member of group 'LOCAL'.
[2013-06-13T13:23:54]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
[2013-06-13T13:23:54]   Member of group 'BUILTIN\Administrators'.
[2013-06-13T13:23:54] Updated Progress 'Validating the parameters.' 2%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] Mailbox server: value passed in = SEMEX06, mailboxServer.Name = SEMEX06, mailboxServer.Fqdn = SEMEX06.sug.com
[2013-06-13T13:23:54] LogClussvcState: clussvc is Stopped on SEMEX06.sug.com. Exception (if any) = none
[2013-06-13T13:23:54] The IP addresses for the DAG are (blank means DHCP): 10.1.5.162
[2013-06-13T13:23:54] Looking up IP addresses for ExchDag.
[2013-06-13T13:23:54]   ExchDag = [ 10.1.5.162 ].
[2013-06-13T13:23:54] Looking up IP addresses for semex06.
[2013-06-13T13:23:54]   semex06 = [ 10.1.5.160 ].
[2013-06-13T13:23:54] Looking up IP addresses for SEMEX06.sug.com.
[2013-06-13T13:23:54]   SEMEX06.sug.com = [ 10.1.5.160 ].
[2013-06-13T13:23:54] DAG ExchDag has 2 servers:
[2013-06-13T13:23:54] DAG ExchDag contains server SEMEX05.
[2013-06-13T13:23:54] DAG ExchDag contains server SEMEX04.
[2013-06-13T13:23:54] Updated Progress 'Checking if Mailbox server 'SEMEX06' is in a database availability group.' 4%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(SEMEX06.sug.com) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed... This is OK.
[2013-06-13T13:23:54] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2013-06-13T13:23:54] DumpClusterTopology: Opening remote cluster ExchDag.
[2013-06-13T13:23:54] Dumping the cluster by connecting to: ExchDag.
[2013-06-13T13:23:54] The cluster's name is: ExchDag.
[2013-06-13T13:23:54] Groups
[2013-06-13T13:23:54]     group: Available Storage [not a CMS]
[2013-06-13T13:23:54]         OwnerNode: SEMEX04.sug.com
[2013-06-13T13:23:54]         State: Offline
[2013-06-13T13:23:54]     group: Cluster Group [Cluster Main Group]
[2013-06-13T13:23:54]         OwnerNode: SEMEX05.sug.com
[2013-06-13T13:23:54]         State: Online
[2013-06-13T13:23:54]             Resource: IPv4 Static Address 1 (Cluster Group) [Online, type = IP Address, PossibleOwners = SEMEX04,SEMEX05 ]
[2013-06-13T13:23:54]                 Address = [10.1.5.162]
[2013-06-13T13:23:54]                     EnableDhcp = [0]
[2013-06-13T13:23:54]                     Network = [Cluster Network 1]
[2013-06-13T13:23:54]             Resource: Cluster Name [Online, type = Network Name, PossibleOwners = SEMEX04,SEMEX05 ]
[2013-06-13T13:23:54]                 NetName = [EXCHDAG]
[2013-06-13T13:23:54]                     RequireKerberos = 1. Computer account created on DC ''.
[2013-06-13T13:23:54]             Resource: File Share Witness (\\semfs01.sug.com\ExchDag.sug.com) [Online, type = File Share Witness, PossibleOwners = SEMEX04,SEMEX05 ]
[2013-06-13T13:23:54] Nodes
[2013-06-13T13:23:54]     node: SEMEX04.sug.com [ state = Up ]
[2013-06-13T13:23:54]     node: SEMEX05.sug.com [ state = Up ]
[2013-06-13T13:23:54] Subnets
[2013-06-13T13:23:54]     Name(Cluster Network 1), Mask(10.1.0.0/16), Role(ClusterNetworkRoleInternalAndClient)
[2013-06-13T13:23:54]         NIC 10.1.5.159 on Node SEMEX05 in State=Up
[2013-06-13T13:23:54]         NIC 10.1.5.158 on Node SEMEX04 in State=Up
[2013-06-13T13:23:54]     Name(Cluster Network 3), Mask(10.150.0.0/16), Role(ClusterNetworkRoleInternalAndClient)
[2013-06-13T13:23:54]         NIC 10.150.0.120 on Node SEMEX04 in State=Up
[2013-06-13T13:23:54]         NIC 10.150.0.122 on Node SEMEX05 in State=Up
[2013-06-13T13:23:54] Opening the cluster on nodes [semex05, semex04].
[2013-06-13T13:23:54] Other mailbox servers in the DAG are already members of cluster 'ExchDag'
[2013-06-13T13:23:54] The server SEMEX06 does not belong to a cluster, and the other servers belong to ExchDag.
[2013-06-13T13:23:54] Successfully resolved the servers based on the stopped servers list.
[2013-06-13T13:23:54] The following servers are in the StartedServers list (The list is the StartedServers property of the DAG in AD):
[2013-06-13T13:23:54] The following servers are in the StoppedServers list:
[2013-06-13T13:23:54] Verifiying that the members of database availability group 'ExchDag' are also members of the cluster.
[2013-06-13T13:23:54] Verifying that the members of cluster 'ExchDag' are also members of the database availability group.
[2013-06-13T13:23:54] According to GetNodeClusterState(), the server SEMEX06 is NotConfigured.
[2013-06-13T13:23:54] The CNO is currently Online.
[2013-06-13T13:23:54] InternalValidate() done.
[2013-06-13T13:23:54] Updated Progress 'Adding server 'SEMEX06' to database availability group 'ExchDag'.' 6%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] Updated Progress 'Adding server 'SEMEX06' to the cluster.' 8%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:30:17] The following log entry comes from a different process that's running on machine 'SEMEX05.sug.com'. BEGIN
[2013-06-13T13:30:17] [2013-06-13T13:23:54] Opening a local AmCluster handle.
[2013-06-13T13:23:54] Updated Progress 'Adding server 'semex06' to database availability group 'ExchDag'.' 2%.
[2013-06-13T13:23:54] Working
[2013-06-13T13:23:54] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:54] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:54] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = ExchDag, dwStatus = 0x0 )
[2013-06-13T13:23:55] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:55] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:56] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = SEMEX06.sug.com, dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = , dwStatus = 0x0 )
[2013-06-13T13:23:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:26:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 100, szObjectName = SEMEX06, dwStatus = 0x5b4 )
[2013-06-13T13:26:57] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = SEMEX06, dwStatus = 0x0 )
[2013-06-13T13:30:17] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = , dwStatus = 0x0 )

[2013-06-13T13:30:17] The preceding log entry comes from a different process running on computer 'SEMEX05.sug.com'. END
[2013-06-13T13:30:17] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-06-13_13-23-54.076_add-databaseavailabiltygroupserver.log".
[2013-06-13T13:30:17] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed.. ---> System.ComponentModel.Win32Exception: This operation returned because the timeout period expired
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
   --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) ---
   at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
   at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
   --- End of stack trace on server (SEMEX05.sug.com) ---
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(AmServerName serverName, String databaseName, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(AmServerName serverName, Nullable`1 dbGuid, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
[2013-06-13T13:30:17] Updated Progress 'Done!' 100%.
[2013-06-13T13:30:17] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().
0
 
LVL 4

Assisted Solution

by:Alexander Kireev
Alexander Kireev earned 500 total points
ID: 39258678
Hello,

Do your DAG members use two network adapters (first for DAG, second for data)?
Do you sure that you haven't issues with ip-addresses? Could you send a screenshot of your DAG networks (Organization... - Mailbox - DAG - Networks) and ipconfig of new server?

Are SEMEX06 and other DAG members has same Mailbox/HUB role or different roles?
Is SEMEX06 has the same Exchange version (include RU)?

Send me a result of cmdlt "Get-DatabaseAvailabilityGroup | fl".

http://exchangeserverpro.com/exchange-server-2010-database-availability-group-installation-step-by-step/
0
 

Accepted Solution

by:
jonasdurst earned 0 total points
ID: 39327976
Since we are going with an entirely new domain I'm rebuilding the exchange environment from the ground up, no longer have a need to add the additional MDB.   I have physical servers, going to build two CAS/HUB servers and then two MDB servers.
0
 

Author Closing Comment

by:jonasdurst
ID: 39345319
Since I've avoided the issue buy going down another road I technically didn't solve the issue
0

Featured Post

Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

As with any other System Center product, the installation for the Authoring Tool can be quite a pain sometimes. This article serves to help you avoid making these mistakes and hopefully save you a ton of time on troubleshooting :)  Step 1: Make sur…
Find out what you should include to make the best professional email signature for your organization.
The viewer will learn how to simulate a series of sales calls dependent on a single skill level and learn how to simulate a series of sales calls dependent on two skill levels. Simulating Independent Sales Calls: Enter .75 into cell C2 – “skill leve…
This video demonstrates how to sync Microsoft Exchange Public Folders with smartphones using CodeTwo Exchange Sync and Exchange ActiveSync. To learn more about CodeTwo Exchange Sync and download the free trial, go to: http://www.codetwo.com/excha…

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question