Exchange 2010 Database Availability Group Error

Do you have all the prereq's installed? Especially Clustering?
RSAT-Clustering
http://blog.exchangegeek.com/2009/07/exchange-2010-windows-server-2008-r2.html

Can you post the dagtask log add-databaseavailabiltygroupserver.log?

I believe my issue is with using vmware vsphere template to deploy the 2008 R2 box.  Going to try to run sysprep and see if that resolves my problem.

Still getting the same error.  Here is the detailed log file.  Any ideas? (Prereqs are okay)

add-databaseavailabiltygroupserver started on machine BRADTESTSRV1.
[2009-12-28T17:03:52] add-dagserver started
[2009-12-28T17:03:52] commandline:         $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2009-12-28T17:03:52] Option 'Identity' = 'DAG1'.
[2009-12-28T17:03:52] Option 'MailboxServer' = 'BradTestSrv1'.
[2009-12-28T17:03:52] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2009-12-28T17:03:52] Option 'WhatIf' = ''.
[2009-12-28T17:03:52] Process: w3wp w3wp.exe:3136.
[2009-12-28T17:03:52] User context = 'NT AUTHORITY\SYSTEM'.
[2009-12-28T17:03:52]   Member of group 'Everyone'.
[2009-12-28T17:03:52]   Member of group 'BUILTIN\Users'.
[2009-12-28T17:03:52]   Member of group 'NT AUTHORITY\SERVICE'.
[2009-12-28T17:03:52]   Member of group 'CONSOLE LOGON'.
[2009-12-28T17:03:52]   Member of group 'NT AUTHORITY\Authenticated Users'.
[2009-12-28T17:03:52]   Member of group 'NT AUTHORITY\This Organization'.
[2009-12-28T17:03:52]   Member of group 'BUILTIN\IIS_IUSRS'.
[2009-12-28T17:03:52]   Member of group 'LOCAL'.
[2009-12-28T17:03:52]   Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
[2009-12-28T17:03:52]   Member of group 'BUILTIN\Administrators'.
[2009-12-28T17:03:52] Updated Progress 'Validating the parameters.' 2%.
[2009-12-28T17:03:52] Working
[2009-12-28T17:03:53] Mailbox server: value passed in = BradTestSrv1, mailboxServer.Name = BRADTESTSRV1, mailboxServer.Fqdn = BradTestSrv1.utopia.local
[2009-12-28T17:03:53] LogClussvcState: clussvc is Stopped on BradTestSrv1.utopia.local. Exception (if any) = none
[2009-12-28T17:03:53] The IP addresses for the DAG are (blank means DHCP): 172.31.8.13
[2009-12-28T17:03:53] Looking up IP addresses for DAG1.
[2009-12-28T17:03:53]   DAG1 = [ 172.31.8.13 ].
[2009-12-28T17:03:53] Looking up IP addresses for bradtestsrv1.
[2009-12-28T17:03:53]   bradtestsrv1 = [ fe80::7d5f:6c38:d2e2:b6d4%18, 172.31.8.10 ].
[2009-12-28T17:03:53] Looking up IP addresses for BradTestSrv1.utopia.local.
[2009-12-28T17:03:53]   BradTestSrv1.utopia.local = [ fe80::7d5f:6c38:d2e2:b6d4%18, 172.31.8.10 ].
[2009-12-28T17:03:53] DAG DAG1 has 1 servers:
[2009-12-28T17:03:53] DAG DAG1 contains server BRADTESTSRV3.
[2009-12-28T17:03:53] Updated Progress 'Checking if Mailbox server 'BRADTESTSRV1' is in a database availability group.' 4%.
[2009-12-28T17:03:53] Working
[2009-12-28T17:03:53] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(BradTestSrv1.utopia.local) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed.. This is OK.
[2009-12-28T17:03:53] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2009-12-28T17:03:53] DumpClusterTopology: Opening remote cluster DAG1.
[2009-12-28T17:03:53] Dumping the cluster by connecting to: DAG1.
[2009-12-28T17:03:53] The cluster's name is: DAG1.
[2009-12-28T17:03:53] Groups
[2009-12-28T17:03:53]     group: Available Storage [not a CMS]
[2009-12-28T17:03:53]         OwnerNode: BRADTESTSRV3.utopia.local
[2009-12-28T17:03:53]         State: Offline
[2009-12-28T17:03:53]     group: Cluster Group [Cluster Main Group]
[2009-12-28T17:03:53]         OwnerNode: BRADTESTSRV3.utopia.local
[2009-12-28T17:03:53]         State: Online
[2009-12-28T17:03:53]             Resource: Cluster IP Address [Online, type = IP Address, PossibleOwners = BRADTESTSRV3 ]
[2009-12-28T17:03:53]                 Address = [172.31.8.13]
[2009-12-28T17:03:53]                     EnableDhcp = [0]
[2009-12-28T17:03:53]                     Network = [Cluster Network 1]
[2009-12-28T17:03:53]             Resource: Cluster Name [Online, type = Network Name, PossibleOwners = BRADTESTSRV3 ]
[2009-12-28T17:03:53]                 NetName = [DAG1]
[2009-12-28T17:03:53] Nodes
[2009-12-28T17:03:53]     node: BRADTESTSRV3.utopia.local [ state = Up ]
[2009-12-28T17:03:53] Subnets
[2009-12-28T17:03:54]     Name(Cluster Network 1), Mask(172.31.8.0/24), Role(ClusterNetworkRoleInternalAndClient)
[2009-12-28T17:03:54]         NIC 172.31.8.12 on Node BRADTESTSRV3 in State=Up
[2009-12-28T17:03:54] Opening the cluster on nodes [bradtestsrv3].
[2009-12-28T17:03:54] Other mailbox servers in the DAG are already members of cluster 'DAG1'
[2009-12-28T17:03:54] The server BRADTESTSRV1 does not belong to a cluster, and the other servers belong to DAG1.
[2009-12-28T17:03:54] Successfully resolved servers based on the AD configuration.
[2009-12-28T17:03:54] The following servers are in the StartedServers list:
[2009-12-28T17:03:54] The following servers are in the StoppedServers list:
[2009-12-28T17:03:54] Verifiying that the members of database availability group 'DAG1' are also members of the cluster.
[2009-12-28T17:03:54] Verifying that the members of cluster 'DAG1' are also members of the database availability group.
[2009-12-28T17:03:54] According to GetNodeClusterState(), the server BRADTESTSRV1 is NotConfigured.
[2009-12-28T17:03:54] The CNO is currently Online.
[2009-12-28T17:03:54] InternalValidate() done.
[2009-12-28T17:03:54] Updated Progress 'Adding server 'BRADTESTSRV1' to database availability group 'DAG1'.' 6%.
[2009-12-28T17:03:54] Working
[2009-12-28T17:03:54] Updated Progress 'Adding server 'BRADTESTSRV1' to the cluster.' 8%.
[2009-12-28T17:03:54] Working
[2009-12-28T17:13:51] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2009-12-28_17-03-52.477_add-databaseavailabiltygroupserver.log".
[2009-12-28T17:13:51] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed with message: Error 0x71a (The remote procedure call was cancelled) from cli_RpccAddNodeToCluster ---> Microsoft.Exchange.Rpc.RpcException: Error 0x71a (The remote procedure call was cancelled) from cli_RpccAddNodeToCluster
   at ThrowRpcException(Int32 rpcStatus, String message)
   at Microsoft.Exchange.Rpc.RpcClientBase.ThrowRpcException(Int32 rpcStatus, String routineName)
   at Microsoft.Exchange.Rpc.Cluster.ReplayRpcClient.RpccAddNodeToCluster(String newNode, String& verboseLog)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.<>c__DisplayClass26.<RunAddNodeToCluster>b__24()
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.<>c__DisplayClass32.<RunRpcOperationDbName>b__30()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRetryableOperation(String serverName, RpcClientOperation rpcOperation)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientHandleRpcException(RpcException ex, String serverName)
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRetryableOperation(String serverName, RpcClientOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(String serverName, String databaseName, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(String serverName, Nullable`1 dbGuid, Int32 timeoutMs, ReplayRpcClient& rpcClient, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
   at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
   at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
[2009-12-28T17:13:51] Updated Progress 'Done!' 100%.
[2009-12-28T17:13:51] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

Please verify:
1. The operating systems are exactly the same on each node being added to the DAG (no mix of R2 vs non-R2; no mix of SP2 vs non-SP2, no mix of Enterprise vs Standard, etc)
2. The same service packs and exact same windows updates patches are applied
3. The MAC/self-assigned IP addresses of your NICs and pseudo-devices are different between the two VM servers (If you are using a Gold image/vm template!)

I had to disable Teredo tunneling during a similar situation... [When examining two different instances of an image from master image W2K8x64_gold_image, the two different Teredo pseudo-interfaces had the same self-assigned v6 ip address!!! Since the Cluster Role was installed on my Gold-image before the template was created, we could uninstall Cluster Role and reinstall it on each node perhaps...? I need to test this in the lab]
      I ended up doing this:
      Disable Teredo through command line
      a.       Open ‘Command Prompt’ and ‘Run as Administrator’
      b.      Type:
            i.      > netsh
            ii.      > interface
            iii.      > teredo
            iv.      > set state disabled
      c.       Teredo will now be disabled

      Also, I Uninstalled the Teredo Tunneling Pseudo-device using Device Manager; but I had to 'Show Hidden Devices' within Device Manager in order to 'see' it.
Ref: http://technet.microsoft.com/en-us/library/cc770710(WS.10).aspx


Disable/remove Teredo on both servers, and try adding your cluster node again to the DAG.

Some other things to try:
4. Please check the NIC binding order in ‘Advanced Settings’ under ‘Network’.
5. Please try to ping the cluster IP address and Cluster name from the problematic machine.
6. look in the events logs on both nodes for any Windows Install errors or any Cluster related errors

I also had this problem and Symantec Endpoint Protection was the issue. However I did not uninstall the product. I just turned off "Network Threat Protection" and the second server was added with no errors.