SYSVOL Not Replicating

We have 2 Windows Server 2008 R2 domain controllers, one PDC and a secondary which are not replicating the SYSVOL. Everything appears fine with AD, as all users, computers and DNS settings are replicating correctly. The DC’s are separated by a firewall. Here is a recap of the problem.

1.      Created a new OU in AD and moved the PDC into it.
2.      Created a new GPO and when we ran gpupdate /force received an error
3.      Noticed that the Sysvol was not replicating between DC’s
4.      Moved the PDC back to the original OU
5.      Now gpupdate /force runs without any errors.
6.      Sysvol is still not replicating.

The event log on both DC’s have the following error: The DFS Replication service failed to communicate with partner (other DC) for replication group Domain System Volume. This error can occur if the host is unreachable, or if the DFS Replication service is not running on the server.
Additional Information:
Error 1722 (The RPC server is unavailable)

Verified start-up value and service status is correct for the RPC (Started/Automatic), RPC Locator (Not started Manual) and Kerberos Key Distribution Center Started Automatic)

Verified the ClientProtocols key exists under HKLM\Software\Microsoft\Rpc and that it contains the correct default protocols.
Here is a list of the ports open between the two DC’s.

tcp destination eq 135
udp destination eq 135
tcp destination eq 137
udp destination eq netbios-ns
udp destination eq netbios-dgm
tcp destination eq netbios-ssn
udp destination eq 389
tcp destination eq 445
tcp destination eq 88
tcp destination eq domain
udp destination eq domain
udp destination eq ntp
udp destination eq 88
tcp destination eq 1025
tcp destination eq ldap
udp destination eq 445
tcp destination eq ldaps
tcp destination eq 3268
tcp destination eq 3269
tcp destination eq 1026
tcp destination eq 1272
tcp destination eq 1190
tcp destination eq 1053
tcp destination eq 464
udp destination eq 46
tcp destination range 49152 63999

As a last ditch effort we forcefully demoted the Secondary DC using DCPromo and then cleaned up the metadata. We then promoted it again using DCPromo. Same results, event log on both DC’s show same Error 1722 noted above and in addition, the Event Log of the Secondary DC has an event stating that “The DFS Replication service initialized SYSVOL at local path C:\Windows\SYSVOL\domain and is waiting to perform initial replication. The replicated folder will remain in the initial synchronization state until it has replicated with its partner (the PDC).

DCDIAG runs clean on the PDC, as does repadmin /syncall.
Repadmin /replsum does not show any errors.
Repadmin /showrepl shows all the Last attempts were successful.

All of the above tests also ran clean on the Secondary DC before we demoted and promoted it.

Appreciate any suggestions.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Brad BouchardInformation Systems Security OfficerCommented:
The first and only thing you did wrong was to move any Domain Controller out of the Domain Controllers OU.  It has settings on its GPO that is only applied to DCs in that OU.  Move it back and then just add the GPO with the settings you want as a separate linked GPO.  Never move domain controllers from their default containers.
Brad BouchardInformation Systems Security OfficerCommented:
Also, what I'd HIGHLY recommend is that you start from scratch with a new, or multiple new, domain controllers.  Demote, and then unjoin the problematic ones from the domain and then remove them from Sites and Services as well as any DNS traces.  After this, recreate new DCs and join them to the domain, static IP them, then promote to DC.

Let me know if you still need help.
TomProAuthor Commented:
We did previously move the Domain Controller back, but still have SYSVOL replication issue. These are the only 2 Domain Controllers in the domain and until I can get the PDC to replicate the SYSVOL with another DC, it is the only fully functioning one in the domain.
Cloud Class® Course: CompTIA Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

Brad BouchardInformation Systems Security OfficerCommented:
Try this... then let me know...

Specifically the section under "Detailed list of steps"
Brad BouchardInformation Systems Security OfficerCommented:
And actually, before you try my article above, try manually replicating from Sites and Services.  Do you know how to do this?
TomProAuthor Commented:
Yes, I have manually replicated from sites and services several times. Replicates users and computer perfectly when initiated from either Domain Controller, Problem seems to be just the SYSVOL does not replicate. The article above references FRS and our DC's are Windows server 2008 R2 using DFS. Is there a comparable procedure for DFS?
Brad BouchardInformation Systems Security OfficerCommented:
You can do the same procedure for 2008 R2.  Don't worry about the FRS.  That's for if you have 2003 DCs still in your domain.  Try the above steps then let me know what happens.  If all else fails can you try adding a 3rd DC to see if it promotes ok?  Then give all FSMOs to that DC and demote/unjoin your other two.  Then recreate another DC and things should be ok.  I don't mean to sound bossy but it really is an easier much simpler scenario than trying to fix your replication issues with the current DCs.  My fear in things like this is that there will always be fallout from when you did this (move DC out of default OU) that creeps its head up out of nowhere 6 months down the road.
TomProAuthor Commented:
Thank you for the advice. I will try the above steps to rebuild the SYSVOL and post back the results.
TomProAuthor Commented:
As a point of clarification: we didn't actually remove the DC from the Domain Controllers group, we created a sub-OU under the builtin Domain Controllers OU, and put one of the two DC's in that sub-OU, and allowed inheritance for all functionality in that sub-OU.

In our architecture, we will eventually have 2 global DC's and then a bunch of site DC's which will be local DCs at each of the many physical locations that this domain will reside.
The sub-OU was going to be a means to differentiate which are the global and which are the site specific.

So is this still likely the problem?
We've done this same design in previous W2003 AD architectures, and it didn't cause problems like this one did, so maybe there's something unique about W2008 R2 infrastructure that doesn't like that.
Brad BouchardInformation Systems Security OfficerCommented:
I can't say for 100% sure, but all the more experienced guys that I look to for things like this all say the same thing, and that is to never move a dc regardless if it's just a sub OU.

I am willing to put money on the fact that that is what the problem is.  Did my trick above work?
TomProAuthor Commented:
The article you referenced is definitely geared towards 2003 using FRS, not 2008 using DFS. Most of the registry subkey’s it references do not exist in the registry. The ntfrsutl command is not recognized, nor is the Linkd command. I referenced the detailed steps using DFS commands where possible. Here is a recap:

1.      On a single domain controller, configure the SYSVOL replica set to be authoritative. BurFlags registry entries do not exist, but I verified the PDC was set to be authoritative.
2.      I verified that the PDC has all the required folders and reparse points.
3.      On the reference domain controller, build a good set of policies and scripts, and then put them in a temporary folder outside the SYSVOL replica set folders on the FRS reference domain controller.
To complete this step, examine Active Directory to determine the group policies that are still used and that contain orphaned data. We do not have any group policies which have orphaned data.
4.      BurFlags registry entries do not exist, but I verified the backup DC was set to be non-authoritative.

This article did not provide much help as the major steps, setting burflags is not relevant, as well as there were no policies with orphaned data. The folder structure of the SYSVOL is correct and all reparse points are correct.
Brad BouchardInformation Systems Security OfficerCommented:
Then promote a 3rd DC give it all FSMOs and cleanly remove your problem DC(s), and re-add 1 or 2 more DCs and distribute FSMOs as you see fit.  Easiest way to do it.  Also be sure to leave it in the default container.
TomProAuthor Commented:
After all this, the issue ended up being a closed port on the firewall. Port 5722 was the culprit. As you can see by my original posting, we did not have this port open, as we were not aware it was used by DFS.
The Microsoft DFS Replication Service is the protocol registered with IANA that uses the communication port 5722. This protocol is associated with the DFS (Distributed File System) technology introduced by Microsoft Corporation under the Microsoft Windows Server 2003 Release 2.

xBouchardx, thank you for taking the time to respond with your advise and suggestions. It was very much appreciated.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Brad BouchardInformation Systems Security OfficerCommented:
No problem, I'm glad you figured out what the issue was.  Very strange indeed.
TomProAuthor Commented:
Identified closed port which was blocking DFS from replicating sysvol
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Legacy OS

From novice to tech pro — start learning today.