Password Hash Synchronization sporadically failing with event 611, unable to retrieve source domain information

I have been having an issue with Azure AD Connect and have run out of troubleshooting ideas and hoping someone can help. I have a support ticket open with our Office 365 re-seller but so far they have been unwilling to open a ticket with Microsoft.

The issue is that our password hash synchronization stops working after a few days. The event viewer logs an event 611, Password hash synchronization failed for domain: (our domain name). Details:
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: (our domain name). Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

I originally uninstalled and reinstalled Azure AD Connect and that fixed the problem for a few days. Then the same error re-occurred. I originally thought that it might have been a networking issue with our Hyper-V cluster because disabling IP-Sec offloading on the VM and rebooting the VM caused the password synchronization to start working. However, I think that was just a coincidence because the password hash failed again a day later and doing the same thing had no effect the second time around. Either way, the support agent I have been working with wanted me to install the Azure AD Connect on a physical computer, which I did. Again, password sync worked for a few days and then started failing. Interestingly, it would stop working in the middle of the night and start working again for a few days and then just last night it stopped working again. The support agent I was working with thought it was because I didn't have the necessary groups in AD, (ADSyncAdmins, ADSyncBrowse, ADSyncOperators, ADSyncPasswordSet). He had me make the groups in AD (even though the documentation clearly states that they can be local groups on the computer where Azure AD Connect is installed) and now wants me to uninstall and reinstall Azure AD Connect and point the install to use those groups instead of creating them locally. I will do this but am very skeptical that this is the underlying problem. If the groups were the problem the password sync would not work at all instead of being a sporadic issue.

We have been in the process of upgrading our domain controllers to Server 2016 with our physical domain controller just being upgraded to Server 2012 from 2008 R2. By upgraded I mean building a new server, migrating roles and then demoting the old domain controller. The password hash issue started appearing after the demoting of our old physical domain controller but don't know if there could be a correlation. We have a Palo Alto firewall and I have combed the traffic logs as best as I could thinking maybe the firewall is blocking something but can't find anything unusual. If anyone has seen this issue before and knows what it could be I would greatly appreciate some help. Below is the full error in the event log:

Password hash synchronization failed for domain: domain.local. Details:
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: domain.local. Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.RefreshCache()
   at System.DirectoryServices.DirectoryEntry.FillCache(String propertyName)
   at System.DirectoryServices.DirectoryEntry.get_NativeGuid()
   at System.DirectoryServices.DirectoryEntry.get_Guid()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.ReadServerGuids(SourceDomainController sourceDomainInfo)
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.CreateSourceDomainInformation()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.Connect()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.<>c__DisplayClass2_0.<ExecuteWithRetry>b__0()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.ExecuteWithRetry[T](Func`1 operation, Func`1 shouldAbort, RetryPolicyHandler retryPolicy)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.CreateConnection()
   at Microsoft.Online.PasswordSynchronization.DeltaSynchronizationTask.SynchronizeCredentialsToCloud()
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.SynchronizeSecrets()
   at Microsoft.Online.PasswordSynchronization.SynchronizationExecutionContext.SynchronizeDomain()
   at Microsoft.Online.PasswordSynchronization.SynchronizationManager.SynchronizeDomain(SynchronizationExecutionContext syncExecutionContext)
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: domain.local. Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.RefreshCache()
   at System.DirectoryServices.DirectoryEntry.FillCache(String propertyName)
   at System.DirectoryServices.DirectoryEntry.get_NativeGuid()
   at System.DirectoryServices.DirectoryEntry.get_Guid()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.ReadServerGuids(SourceDomainController sourceDomainInfo)
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.CreateSourceDomainInformation()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.Connect()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.<>c__DisplayClass2_0.<ExecuteWithRetry>b__0()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.ExecuteWithRetry[T](Func`1 operation, Func`1 shouldAbort, RetryPolicyHandler retryPolicy)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.CreateConnection()
   at Microsoft.Online.PasswordSynchronization.DeltaSynchronizationTask.SynchronizeCredentialsToCloud()
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.SynchronizeSecrets()
   at Microsoft.Online.PasswordSynchronization.SynchronizationExecutionContext.SynchronizeDomain()
   at Microsoft.Online.PasswordSynchronization.SynchronizationManager.SynchronizeDomain(SynchronizationExecutionContext syncExecutionContext)
Microsoft.Online.PasswordSynchronization.SynchronizationManagerException: Unable to open connection to domain: domain.local. Error: Unable to retrieve source domain information. ---> Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsException: Unable to retrieve source domain information. ---> System.DirectoryServices.DirectoryServicesCOMException: There is no such object on the server.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.RefreshCache()
   at System.DirectoryServices.DirectoryEntry.FillCache(String propertyName)
   at System.DirectoryServices.DirectoryEntry.get_NativeGuid()
   at System.DirectoryServices.DirectoryEntry.get_Guid()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.ReadServerGuids(SourceDomainController sourceDomainInfo)
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.CreateSourceDomainInformation()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.EstablishConnection()
   at Microsoft.Online.PasswordSynchronization.DirectoryReplicationServices.DrsConnection.Connect()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.<>c__DisplayClass2_0.<ExecuteWithRetry>b__0()
   at Microsoft.Online.PasswordSynchronization.RetryUtility.ExecuteWithRetry[T](Func`1 operation, Func`1 shouldAbort, RetryPolicyHandler retryPolicy)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   --- End of inner exception stack trace ---
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.OpenConnection(IDrsConnection connection)
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.CreateConnection()
   at Microsoft.Online.PasswordSynchronization.DeltaSynchronizationTask.SynchronizeCredentialsToCloud()
   at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.SynchronizeSecrets()
   at Microsoft.Online.PasswordSynchronization.SynchronizationExecutionContext.SynchronizeDomain()
   at Microsoft.Online.PasswordSynchronization.SynchronizationManager.SynchronizeDomain(SynchronizationExecutionContext syncExecutionContext)
.

<forest-info>
  <forest-name>domain.local</forest-name>
  <connector-id>0bd53522-21e7-4905-9f63-f25690b63e07</connector-id>
</forest-info>
Adam KaczorAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ayesh JuniorCommented:
I would check connectors -- Click on On-Prem connector -- Properties -- click on Configure Partitions ... -- Check if you have added a domain name in this window instead of a domain controller FQDN
Adam KaczorAuthor Commented:
Hi Ayesh, yes I have it set to use a domain name and not a FQDN of a domain controller.
Ayesh JuniorCommented:
Hi Adam,

I'm truly sorry for replying so late, you will need to use FQDN of your domain controller and not your domain name or leave this field empty.
Adam KaczorAuthor Commented:
I figured out the issue. Turns out it was our Open DNS forwarder. I had the DNS set to use the forwarder and not the domain controllers.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Adam KaczorAuthor Commented:
The VM was pointing to the Open DNS forwarder for the DNS. Switching to the domain controllers as DNS solved the problem.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Office 365

From novice to tech pro — start learning today.