Password Hash Synchronization sporadically failing with event 611, unable to retrieve source domain information

Adam Kaczor
Adam Kaczor used Ask the Experts™
on
I have been having an issue with Azure AD Connect and have run out of troubleshooting ideas and hoping someone can help. I have a support ticket open with our Office 365 re-seller but so far they have been unwilling to open a ticket with Microsoft.

The issue is that our password hash synchronization stops working after a few days. The event viewer logs an event 611, Password hash synchronization failed for domain: (our domain name). Details:
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: (our domain name). Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

I originally uninstalled and reinstalled Azure AD Connect and that fixed the problem for a few days. Then the same error re-occurred. I originally thought that it might have been a networking issue with our Hyper-V cluster because disabling IP-Sec offloading on the VM and rebooting the VM caused the password synchronization to start working. However, I think that was just a coincidence because the password hash failed again a day later and doing the same thing had no effect the second time around. Either way, the support agent I have been working with wanted me to install the Azure AD Connect on a physical computer, which I did. Again, password sync worked for a few days and then started failing. Interestingly, it would stop working in the middle of the night and start working again for a few days and then just last night it stopped working again. The support agent I was working with thought it was because I didn't have the necessary groups in AD, (ADSyncAdmins, ADSyncBrowse, ADSyncOperators, ADSyncPasswordSet). He had me make the groups in AD (even though the documentation clearly states that they can be local groups on the computer where Azure AD Connect is installed) and now wants me to uninstall and reinstall Azure AD Connect and point the install to use those groups instead of creating them locally. I will do this but am very skeptical that this is the underlying problem. If the groups were the problem the password sync would not work at all instead of being a sporadic issue.

We have been in the process of upgrading our domain controllers to Server 2016 with our physical domain controller just being upgraded to Server 2012 from 2008 R2. By upgraded I mean building a new server, migrating roles and then demoting the old domain controller. The password hash issue started appearing after the demoting of our old physical domain controller but don't know if there could be a correlation. We have a Palo Alto firewall and I have combed the traffic logs as best as I could thinking maybe the firewall is blocking something but can't find anything unusual. If anyone has seen this issue before and knows what it could be I would greatly appreciate some help. Below is the full error in the event log:

Password hash synchronization failed for domain: domain.local. Details:
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: domain.local. Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.RefreshCache()
   at System.DirectoryServices.DirectoryEntry.FillCache(String propertyName)
   at System.DirectoryServices.DirectoryEntry.get_NativeGuid()
   at System.DirectoryServices.DirectoryEntry.get_Guid()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.ReadServerGuids(SourceDomainController sourceDomainInfo)
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.CreateSourceDomainInformation()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.Connect()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.<>c__DisplayClass2_0.<ExecuteWithRetry>b__0()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.ExecuteWithRetry[T](Func`1 operation, Func`1 shouldAbort, RetryPolicyHandler retryPolicy)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.CreateConnection()
   at Microsoft.Online.PasswordSynchronization.DeltaSynchronizationTask.SynchronizeCredentialsToCloud()
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.SynchronizeSecrets()
   at Microsoft.Online.PasswordSynchronization.SynchronizationExecutionContext.SynchronizeDomain()
   at Microsoft.Online.PasswordSynchronization.SynchronizationManager.SynchronizeDomain(SynchronizationExecutionContext syncExecutionContext)
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: domain.local. Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.RefreshCache()
   at System.DirectoryServices.DirectoryEntry.FillCache(String propertyName)
   at System.DirectoryServices.DirectoryEntry.get_NativeGuid()
   at System.DirectoryServices.DirectoryEntry.get_Guid()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.ReadServerGuids(SourceDomainController sourceDomainInfo)
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.CreateSourceDomainInformation()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.Connect()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.<>c__DisplayClass2_0.<ExecuteWithRetry>b__0()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.ExecuteWithRetry[T](Func`1 operation, Func`1 shouldAbort, RetryPolicyHandler retryPolicy)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.CreateConnection()
   at Microsoft.Online.PasswordSynchronization.DeltaSynchronizationTask.SynchronizeCredentialsToCloud()
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.SynchronizeSecrets()
   at Microsoft.Online.PasswordSynchronization.SynchronizationExecutionContext.SynchronizeDomain()
   at Microsoft.Online.PasswordSynchronization.SynchronizationManager.SynchronizeDomain(SynchronizationExecutionContext syncExecutionContext)
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: domain.local. Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.RefreshCache()
   at System.DirectoryServices.DirectoryEntry.FillCache(String propertyName)
   at System.DirectoryServices.DirectoryEntry.get_NativeGuid()
   at System.DirectoryServices.DirectoryEntry.get_Guid()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.ReadServerGuids(SourceDomainController sourceDomainInfo)
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.CreateSourceDomainInformation()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.Connect()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.<>c__DisplayClass2_0.<ExecuteWithRetry>b__0()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.ExecuteWithRetry[T](Func`1 operation, Func`1 shouldAbort, RetryPolicyHandler retryPolicy)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.CreateConnection()
   at Microsoft.Online.PasswordSynchronization.DeltaSynchronizationTask.SynchronizeCredentialsToCloud()
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.SynchronizeSecrets()
   at Microsoft.Online.PasswordSynchronization.SynchronizationExecutionContext.SynchronizeDomain()
   at Microsoft.Online.PasswordSynchronization.SynchronizationManager.SynchronizeDomain(SynchronizationExecutionContext syncExecutionContext)
.

<forest-info>
  <forest-name>domain.local</forest-name>
  <connector-id>0bd53522-21e7-4905-9f63-f25690b63e07</connector-id>
</forest-info>
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
I would check connectors -- Click on On-Prem connector -- Properties -- click on Configure Partitions ... -- Check if you have added a domain name in this window instead of a domain controller FQDN

Author

Commented:
Hi Ayesh, yes I have it set to use a domain name and not a FQDN of a domain controller.
Hi Adam,

I'm truly sorry for replying so late, you will need to use FQDN of your domain controller and not your domain name or leave this field empty.
I figured out the issue. Turns out it was our Open DNS forwarder. I had the DNS set to use the forwarder and not the domain controllers.

Author

Commented:
The VM was pointing to the Open DNS forwarder for the DNS. Switching to the domain controllers as DNS solved the problem.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial