We help IT Professionals succeed at work.

unable to add second member to exchange 2010 cross site DAG

emtechadmin
emtechadmin used Ask the Experts™
on
Hi Experts

I'm in a process of implementing two node DAG accross two sites.  I have installed second exchange server at my active directory site.

Site A - 172.16.1.X , DC- 172.16.1.8 Windows 2008 R2

EXCHN1 - 172.16.1.9 - Windows 2008 R2 SP1 with Ex 2010 Sp1

Site B - 172.16.8.X , ADC - 172.16.7.8 -  Windows 2003 sp2,

EXCHN2 - 172.16.1.9 - Windows 2008 R2 Sp1 with Ex 2010 Sp1

 

Now the problems facing are

1.  Getting a warning message when creating a new dag 'The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server <ServerName>.  I have added exchange trusted subsystem to local  administrators group in witness server. But still same error

2. When trying to add second node to DAG , the wizard failing with an error message(svreen shot attached)  'A Server side database availability group administrative  operation failed. Error the operation failed. Create cluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation Error Cluster API "Add clusternode()(max percentage 100)failed with 0x5b4. Error: This operation returned because  the time out period expired"" failed [server: exchn1.mydomain.local]'

I tried removing and reinstalling failover clustering feature on my second node. but still same issue. Awaiting for valuable suggections to resolve this error Error while adding second node to dag
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®

Author

Commented:
logs from Exchange Setup lgs\ dagtask

add-databaseavailabiltygroupserver started on machine PN20.
[2011-08-04T22:07:46] add-dagserver started
[2011-08-04T22:07:46] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2011-08-04T22:07:46] Option 'Identity' = 'FUJDAG'.
[2011-08-04T22:07:46] Option 'MailboxServer' = 'FUJEXCDR'.
[2011-08-04T22:07:46] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2011-08-04T22:07:46] Option 'WhatIf' = ''.
[2011-08-04T22:07:46] Process: w3wp w3wp.exe:8264.
[2011-08-04T22:07:46] User context = 'NT AUTHORITY\SYSTEM'.
[2011-08-04T22:07:46]   Member of group 'Everyone'.
[2011-08-04T22:07:46]   Member of group 'PN20\SQLServer2005MSSQLUser$PN20$BKUPEXEC'.
[2011-08-04T22:07:46]   Member of group 'PN20\SQLServerMSSQLServerADHelperUser$PN20'.
[2011-08-04T22:07:46]   Member of group 'BUILTIN\Users'.
[2011-08-04T22:07:46]   Member of group 'NT AUTHORITY\SERVICE'.
[2011-08-04T22:07:46]   Member of group 'CONSOLE LOGON'.
[2011-08-04T22:07:46]   Member of group 'NT AUTHORITY\Authenticated Users'.
[2011-08-04T22:07:46]   Member of group 'NT AUTHORITY\This Organization'.
[2011-08-04T22:07:46]   Member of group 'BUILTIN\IIS_IUSRS'.
[2011-08-04T22:07:46]   Member of group 'LOCAL'.
[2011-08-04T22:07:46]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
[2011-08-04T22:07:46]   Member of group 'BUILTIN\Administrators'.
[2011-08-04T22:07:46] Updated Progress 'Validating the parameters.' 2%.
[2011-08-04T22:07:46] Working
[2011-08-04T22:07:46] Mailbox server: value passed in = FUJEXCDR, mailboxServer.Name = FUJEXCDR, mailboxServer.Fqdn =

FUJEXCDR.mydomain.local
[2011-08-04T22:07:46] LogClussvcState: clussvc is Stopped on FUJEXCDR.mydomain.local. Exception (if any) = none
[2011-08-04T22:07:46] The IP addresses for the DAG are (blank means DHCP): 172.16.1.20
[2011-08-04T22:07:46] Looking up IP addresses for FUJDAG.
[2011-08-04T22:07:46]   FUJDAG = [ 172.16.1.20 ].
[2011-08-04T22:07:46] Looking up IP addresses for fujexcdr.
[2011-08-04T22:07:46]   fujexcdr = [ 172.16.7.9 ].
[2011-08-04T22:07:46] Looking up IP addresses for FUJEXCDR.mydomain.local.
[2011-08-04T22:07:46]   FUJEXCDR.mydomain.local = [ 172.16.7.9 ].
[2011-08-04T22:07:46] DAG FUJDAG has 1 servers:
[2011-08-04T22:07:46] DAG FUJDAG contains server PN20.
[2011-08-04T22:07:46] Updated Progress 'Checking if Mailbox server 'FUJEXCDR' is in a database availability group.' 4%.
[2011-08-04T22:07:46] Working
[2011-08-04T22:07:46] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An

error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(FUJEXCDR.mydomain.local) failed with 0x6d9.

Error: There are no more endpoints available from the endpoint mapper"' failed... This is OK.
[2011-08-04T22:07:46] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2011-08-04T22:07:46] DumpClusterTopology: Opening remote cluster FUJDAG.
[2011-08-04T22:07:46] Dumping the cluster by connecting to: FUJDAG.
[2011-08-04T22:07:46] The cluster's name is: FUJDAG.
[2011-08-04T22:07:46] Groups
[2011-08-04T22:07:46]     group: Available Storage [not a CMS]
[2011-08-04T22:07:46]         OwnerNode: PN20.mydomain.local
[2011-08-04T22:07:46]         State: Offline
[2011-08-04T22:07:46]     group: Cluster Group [Cluster Main Group]
[2011-08-04T22:07:46]         OwnerNode: PN20.mydomain.local
[2011-08-04T22:07:46]         State: Online
[2011-08-04T22:07:46]             Resource: IPv4 Static Address 1 (Cluster Group) [Online, type = IP Address, PossibleOwners = PN20 ]
[2011-08-04T22:07:46]                 Address = [172.16.1.20]
[2011-08-04T22:07:46]                     EnableDhcp = [0]
[2011-08-04T22:07:46]                     Network = [Cluster Network 1]
[2011-08-04T22:07:46]             Resource: Cluster Name [Online, type = Network Name, PossibleOwners = PN20 ]
[2011-08-04T22:07:46]                 NetName = [FUJDAG]
[2011-08-04T22:07:46] Nodes
[2011-08-04T22:07:46]     node: PN20.mydomain.local [ state = Up ]
[2011-08-04T22:07:46] Subnets
[2011-08-04T22:07:46]     Name(Cluster Network 1), Mask(172.16.1.0/24), Role(ClusterNetworkRoleInternalAndClient)
[2011-08-04T22:07:46]         NIC 172.16.1.9 on Node PN20 in State=Up
[2011-08-04T22:07:46] Opening the cluster on nodes [pn20].
[2011-08-04T22:07:46] Other mailbox servers in the DAG are already members of cluster 'FUJDAG'
[2011-08-04T22:07:46] The server FUJEXCDR does not belong to a cluster, and the other servers belong to FUJDAG.
[2011-08-04T22:07:46] Successfully resolved the servers based on the stopped servers list.
[2011-08-04T22:07:46] The following servers are in the StartedServers list (The list is the StartedServers property of the DAG in AD):
[2011-08-04T22:07:46] The following servers are in the StoppedServers list:
[2011-08-04T22:07:46] Verifiying that the members of database availability group 'FUJDAG' are also members of the cluster.
[2011-08-04T22:07:46] Verifying that the members of cluster 'FUJDAG' are also members of the database availability group.
[2011-08-04T22:07:46] According to GetNodeClusterState(), the server FUJEXCDR is NotConfigured.
[2011-08-04T22:07:46] The CNO is currently Online.
[2011-08-04T22:07:46] InternalValidate() done.
[2011-08-04T22:07:46] Updated Progress 'Adding server 'FUJEXCDR' to database availability group 'FUJDAG'.' 6%.
[2011-08-04T22:07:46] Working
[2011-08-04T22:07:46] Updated Progress 'Adding server 'FUJEXCDR' to the cluster.' 8%.
[2011-08-04T22:07:46] Working
[2011-08-04T22:14:06] The following log entry comes from a different process that's running on machine 'PN20.mydomain.local'. BEGIN
[2011-08-04T22:14:06] [2011-08-04T22:07:46] Opening a local AmCluster handle.
[2011-08-04T22:07:46] Updated Progress 'Adding server 'fujexcdr' to database availability group 'FUJDAG'.' 2%.
[2011-08-04T22:07:46] Working
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType =

ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = FUJEXCDR, dwStatus =

0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType =

ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus = 0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType =

ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = FUJDAG, dwStatus = 0x0

)
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType =

ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = , dwStatus = 0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType =

ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = FUJEXCDR, dwStatus =

0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType =

ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = , dwStatus = 0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseStart,

ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = FUJEXCDR, dwStatus = 0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseEnd,

ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = , dwStatus = 0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType =

ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = FUJEXCDR, dwStatus =

0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType =

ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = , dwStatus = 0x0 )
[2011-08-04T22:07:46] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType =

ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = FUJEXCDR, dwStatus =

0x0 )
[2011-08-04T22:07:47] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType =

ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = , dwStatus = 0x0 )
[2011-08-04T22:07:47] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType =

ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName =

FUJEXCDR.mydomain.local, dwStatus = 0x0 )
[2011-08-04T22:07:47] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseEnd,

ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = , dwStatus = 0x0 )
[2011-08-04T22:07:47] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseStart,

ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = FUJEXCDR, dwStatus = 0x0 )
[2011-08-04T22:10:48] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseEnd,

ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 100, szObjectName = FUJEXCDR, dwStatus = 0x5b4 )
[2011-08-04T22:10:48] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType =

ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = FUJEXCDR, dwStatus =

0x0 )
[2011-08-04T22:14:06] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd,

ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = , dwStatus = 0x0 )

[2011-08-04T22:14:06] The preceding log entry comes from a different process running on computer 'PN20.mydomain.local'. END
[2011-08-04T22:14:06] The operation wasn't successful because an error was encountered. You may find more details in log file "C:

\ExchangeSetupLogs\DagTasks\dagtask_2011-08-04_22-07-46.078_add-databaseavailabiltygroupserver.log".
[2011-08-04T22:14:06] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side

database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from

incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API

'"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed.

---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while

attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation

returned because the timeout period expired"' failed.. ---> System.ComponentModel.Win32Exception: This operation returned because the

timeout period expired
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr

context, Exception& errorException, Boolean throwExceptionOnFailure)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
   --- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) ---
   at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
   at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
   at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName,

RpcServerOperation rpcOperation)
   --- End of stack trace on server (PN20.mydomain.local) ---
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String

serverName, RpcErrorExceptionInfo errorInfo)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(AmServerName serverName, String databaseName,

Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(AmServerName serverName, Nullable`1 dbGuid, Int32

timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode,

String& verboseLog)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
[2011-08-04T22:14:06] Updated Progress 'Done!' 100%.
[2011-08-04T22:14:06] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

dagtask-2011-08-04-22-07-46-078-.log
1. You can ignore this error as it is a known issue

2. When you created the DAG, did you use the shell so that you could create the multiple IP addresses
New-DatabaseAvailabilityGroup -Name Dag1 -DatabaseAvailabilityGroupIpAddresses 172.16.1.x,172.16.8.x -WitnessDirectory c:\Dag1 -WItnessServer Hub1

Author

Commented:
No.. I used EMC to create DAG and then assigned IP address using shell. I will try this option and update the status.

Author

Commented:
Hi..
Added both nodes to DAG.

I have setup this network (all subnets - Main, DR) using a sonicwall. Now I replaced this one with cisco router. I have changed network configuration. Previous attemp was with single nic.  
Then created DAG using above mentioned way.

Now in  dag networks I have 3 networks
172.16.1.0/24  - Production - AD main site
10.10.1.0/24 - Replication at  main site
172.16.7.0 - DR site

Whether this one is right? My replication subnet on DR site(10.10.2.0/24) not listed in DAG networks of EMC.

Anyway I'm proceeding with Test database replication and let you know the results. thanks