SYSVol replication issue

To me it sounds like there is a replication issue that should have errors in the vent logs to point you in the right direction as to why they DC are not staying in synch. I would recommend exporting and importing the GPOs instead trying to copy the folders from one DC to another http://technet.microsoft.com/en-us/library/ee390958.aspx

What steps have you done to try and troubleshoot the replication issues? Have you ran the dcdiag and repadmin tools to check for any issues?

These are great suggestions and I will try this first thing Monday morning.

This is what I have tried, in the following "primary" refers to the DC that holds the PDC Master and Secondary refers to the second one.

OK I have run the DCDiag and it is stating that there is issues with SYSVOl replication, I also ran the repadmin, the script mentioned by Mahesh. I ended up running that script on both DC's to see what the difference would be. The only;y thing I did not try was exporting out the GPO's from the primary and importing them into the secondary. Not sure if that would be the best option right now or would it do any harm to it to try?

I did check GPMC for both DC's and they both show the same GPO's within the GPMC

Attaching all files from the commands that were run.
PrimaryDC-Diag.txt
PrimaryDC-OrphanedGPOs.txt
Repadmin.txt
SecondaryDC-OrphanedGPOs.txt
Secondary-Diag.txt

I even tried to to start the DC in DRSM and do a restore and rebuild of the SYSVol folder but when I logged in as the remote admin it would not let me delete the SYSVol Folder. So that is not going to work.

Do not restore any folder from backup
Because you don't know from when the issue started and which backup can fix it.
 From orphaned GPOs its looks like there are almost 33 folders in sysvol which are orphaned on secondary DC and six on primary DC
U can delete orphaned GPO folders from secondary, but it will not fix replication issue, it is some thing kind of Journal wrap issue.
In my opinion, demote secondary DC gracefully / forcefully if unable to demote correctly followed by Metadata cleanup and then again run orphaned GPO script on primary DC and find out orphaned GPOs
Then remove those orphaned GPO folders, reboot the server, if GPOs are populating correctly I GPMC and again check with script if it finds any thing.
If you don't find any new orphaned GPOs now, its good to go to build another ADC and check if every thing is working as expected
Do not use same name for new DC as earlier, it might create new issues
I hope Exchange server is not installed on DC being demoted.

No there is no exchange server in the environment. So when I demote this what will happen to those machines that are using the demoted DC as there logon server? Is there a script or command line option to force them all to use the good DC?

Probably you could do this in off business hours to minimize impact.
Ensure that healthy DC is preferred DNS server on client workstations (probably you can do this via DHCP if exists or manual intervention is required), and do this activity within off business hours.
After demotion of server do not forget to remove all NS records, Host(A) records, PTR records, (same as folder) records, SRV records for demoted DC from healthy DC
This will ensure that client will logon to healthy server only even if you need more time to deploy another DC after demotion.

Would it make sense to get another healty DC up and running before doing this, maybe migrate from server 2012 to server 2012R2?

OK
U don't want to get into scenario (Risk) where only 1 DC left for time being
Ideally I would not prefer to introduce new DC in existing environment where both DCs are having problem.
Do not try to install 2012 R2 in 2012 AD, it will require additional configuration and increase complexities
However, you can try below.
U can introduce new ADC on new 2012 member server and do not forget to select healthy DC under additional options in dc promotion wizard so that new DC will be promoted with replica from healthy DC and then try to remove faulty DC followed by cleanup Sysvol mess from new and healthy DC as well with the help of script
http://social.technet.microsoft.com/wiki/contents/articles/20098.setting-up-additional-active-directory-domain-controller-with-windows-server-2012.aspx

I will do as you suggested and I will not add a another DC to the environment. It probably is best not to muddy the waters with another DC until the faulty one is removed and we have a healthy environment.

Ok
for safer side take system state backup of primary DC before you demote secondary DC and keep it in safe place

I will have to schedule this over the next few days, and I will let you know the outcome.

Sorry but demoting a DC and creating a new one to troubleshoot SYSVOL replication issues is a bit overkill in my eyes. There's also no point doing this if the replication issues are on your main DC, as any other DCs you introduce into your environment will most likely just experience the same issue.

Have you tried installing the Distributed File System management tools on your machine and running the diagnostic health and propagation tests and reports? This should give you a bit more insight as to why SYSVOL isn't replicating between your DCs.

If you don't have the tools installed, you can add it by going to the Add Roles and Features wizard through Server Manager > Features > expand Remote Server Administration Tools > Role Administration Tools > expand File Services Tools > tick DFS Management Tools > Install.

Once you have installed it, open the DFS Management console and then click Create Diagnostic Report in the Actions pane on the far right. Follow the instructions in the wizard to create the health and propagation reports then return here with your findings.

Also have a look at the logs in the Applications and Services Logs > DFS Replication area in Event Viewer for an warning/error messages and post them here.

I ran through those and it helped resolve a few issues but it did not give me a place to go with the replication. Here are the files.
Health-Domain-20System-20Volume-11Nov201
Propagation-Domain-20System-20Volume-SYS

Anything in the DFS Replication logs? Make sure you check on both DCs.

Did you run the tests on both DCs as well?

I am attaching the screen shots from the DFS Event viewer and those are the errors that I am trying to resolve. I also thought that the DFS test covered both the DC's, they were both listed on the "included servers" list
ProblemDC-DFS-EventViewer.png
PDCMasterDC-DFS-Event-Viewer.png

Yes I did but I am more then willing to try again, the question would be do I go with the Authoritive or the non-authoritive  option?

This is where I had to hunt around

3.Force Active Directory replication throughout the domain and validate its success on all DCs.
             Getting the right command to force it was not the easiest, and how do you validate it?

4.Start the DFSR service set as authoritative:
              Ok starting the service is easy but to start it as Authoritative?????

The non authoritative restore worked the this time. Not sure why it did not the first time but right now I do not care. Tested the replication with a test txt file and it replicated within a second. I appreciate your help.

Glad to hear it's working now. Strange that the authoritative sync didn't work yet the non-authoritative sync did.

Whatever the case, the answers to your questions are as follows:

3.Force Active Directory replication throughout the domain and validate its success on all DCs.
             Getting the right command to force it was not the easiest, and how do you validate it?

To force replication, I always use the steps outlined in this article: http://technet.microsoft.com/en-us/library/cc816926%28v=ws.10%29.aspx
To verify replication you can use this command: repadmin /showrepl - make sure you run this command in an elevated Command Prompt (i.e. right click on Command Prompt > Run as administrator)
More information regarding verifying replication can be found here: http://technet.microsoft.com/en-us/library/cc794749%28v=ws.10%29.aspx

4.Start the DFSR service set as authoritative:
              Ok starting the service is easy but to start it as Authoritative?????

You only need to do this if you stopped DFSR service while performing the authoritative or non authoritative sync. Did you stop this service while performing the steps in the Microsoft KB? If not, you can safely skip this step.