DFS will not work after server restore, new namespace or replication groups fail

Oh dear! Yet again somebody who restores AD Servers from images. I have one thing to say.....

SYSTEM STATE BACKUPS

You will almost certainly have to trash the server you restored from an image and that wont be all. but its a start.

Never use image backups for AD servers unless they are from like last night AND then they MIGHT work.

Trashing the server is not a good option.  Ideas on how to troubleshoot and resolve this would be welcomed.

You may need to dcpromo to demote it, take it out the domain, add it back in and dcpromo it again. Inbetween those steps you may need to do an AD Clean up on the other server.

Restoreing an AD DC from an image is a no no.... It almost always causes more issues than you can ever hope to solve.

After reading this article: http://exchangeserverpro.com/event-id-2095-and-the-usn-rollback-adventure  I see that I have some serious work to do.

However, I do have a question:  in my case, it would be far easier to do away with DC2 because it was only recently added and there's almost no data on it.  It was pretty much there to be a backup of Active Directory and a secondary DNS server.  Also, there has not been much of any major activity in Active Directory at least it terms of adding or editing users and such in the past 6 days.  So I'm wondering if it would make sense to demote and remove DC2 instead, and then just move forward with the state of AD that's on DC1, the one that was restored from image.

If it matters any, users are still logging in and using DC1 without any problems that I know of at the moment.  Based on the article, I suspect I'd start seeing problems when doing AD functions like adding new users.

Would it be reasonable to demote and then remove DC2 instead, then cleaning up references to it in DNS and Sites and Services?  I could then add a new DC, even naming it something different, like DC3, to avoid any conflicts that might still be around from the old name.  

To be clear, my question now becomes: would I be okay to demote and remove DC2, and use what's current on DC1 (the restored one) as the starting point, adding a new DC to that to become my AD backup?  Rebuilding DC1 would be tough given everything including roaming user profiles are stored there.

I did think of one other option.  These servers are virtual machines and the image I spoke if is actually a snapshot.  We have a snapshot of both DCs taken within one hour of each other 6 days ago right before installing DFS.  I could restore both of them to that point together.  Would that be better?  

There are several other servers on the domain that are not DCs.  They are applicationservers and web servers.  I could copy the shared user files from the DC to one of those before rolling back, using XCOPY to preserve permissions, then copy back after the restore so that no data would be lost.

Would this he a better option?

Advice needed urgently.  Thanks!

This is a bit of a special-purpose network in that it's only purpose is to host an application that's accessed via a thin-client connection.  It's using Graphon's Go-Global software where connections look similar to an RDP-connected client in that each user does get a profile created (roaming profile).  They don't show up as workstations in AD though since they are remote.  There are no locally-connected workstations.  And users are prevented from changing their passwords, so those won't have changed.  There have been no servers added or removed since the snapshots.  There are two web servers and about 6 application servers that are part of the domain.

When you say that "workstations will have updated their domain passwords" are you talking about the internal password that they maintain in the background to connect to AD?  Would I simply need to remove them from the domain and re-add them?  Will I have any problem removing them through normal means (i.e. set to a workgroup and enter domain admin credentials to allow it to remove) then adding back, all with same name?

Are there any other issues I should be concerned with if I roll both DCs back?  Anything to be concerned about with IIS on the web servers?  It looks like it was actually about 13 days now.

Any other suggestions or answers to the above questions before I give this a try tonight?  I was wondering if it might be helpful to remove the member servers from the domain before taking them down.  That way they won't think they are joined when the two DCs are restored.  Thoughts?

As I think about this more, I have one other idea: what about demoting then removing DC2 from the current domain rather than rolling them both back.  Wouldn't that leave just one server with AD and thus no conflicts?  Then I could add it back?  If this is not a good idea, then I suppose the questions posted above are still applicable?

Everything is back up and running!

Here's what I did:
I started by restoring both DCs to the same point in time from snapshots that were made prior to DFS installation 12 days ago.  This gave me a functioning AD and users could login and authenticate via DC1. I took DC2 down to verify that everything worked with just DC1.  Adding new users to AD worked fine.  I was able to restore data to the data folders from a backup made right before the rollback.

As mentioned, the other servers (non DC) would not authenticate, so I had to remove them from the domain, delete their references in AD, and add them back.  I didn't have workstations on this domain since it's all remote access, so I didn't have to worry about those.

The part that was still not quite right though was AD synchronization between the DCs.  Objects added at DC1 would show up in DC2, but not visa versa.  DCPROMO /Q showed replication errors and referenced invalid rollback - UCN and noted in the article I linked to earlier.

Since DC2 didn't have much on it and was primarily just a secondary AD and DNS server, it was a lot easier to just remove and rebuild this one.  DCPROMO failed to demote it, so I used the manual method then manual metadata cleanup of sites and services, DNS, etc. Then I added the server back to the domain.  The issue persisted, so I decided to just decommission that VM and server name and install a new one, DC3, which was quick and easy since these are just VMs.  This time, using DCPROMO to demote DC2 worked fine.  I think I still found some references in Sites and Services and DNS, so I cleaned those up.  

With DC2 gone, DC3 easily replaced it and replication of AD is working beautifully with user object showing up on both and DNS replicating.  I installed DFS on DC1 and DC3 and now have successful replication there too.

All is well...details are provided here for any help they may provide to others in similar situation.