Active Directory Re-Sync

Are these in the same AD site or different site?   If they are in the same site in 2003 then replication should be fast (measured in seconds)
So the box blue screened and you rebooted and now you are having issues?  Check your event logs and the output of dcdiag for clues.
You can initiate replication in sites and services or repadmin /syncall
Thanks
Mike

This is a little bit of a shot in the dark, but worth a shot. Since your Infrastructure master is responsible of maintaining AD objects in updated on your domain controllers, and that the Global Catalogs are the ones that keep that transient data quickly accessible to the DCs, I'm thinking one of your DCs acting as a global catalog should still have enough information (even though slightly outdated), to replicate over to your rebuilt DC.

That said, I would say, trigger a replication from a AD1 (which I'm hoping is a global catalog for you), to your new server, and you should be able to restore most of your AD objects that way.

Before you do that though, I would think you may want to seize the infrastructure master role to AD1.

Since I'm not 100% sure of the answer, I would wait to get a couple more confirmations on this thread before proceeding.
And of course, needless to say, please make sure you have a system state backup of your FSMO role holding DCs.

I would not start seizing roles this early.  That is something done when the DC holding the role is down.
Thanks
Mike

Yes, the AD1 is a Global Catalog holder, as is AD2. I should mention that the vDMC server is living in a Virtual CITRIX XenServer pool. It was the installation of the XenTools program that broke the server. It was SNAPSHOT before the failure as well then reverted. I understand that doing that might have been a bad thing as I cannot snapshot Domain Controllers I guess.  I read that in another thread here.

It's sooo close though.

Mike,
I thought he did say that the server holding that role IS down:
"Everything was operating fine until I applied some updates to the server vDMC and it blue screened."
Unless after the blue screen, the server came back up, in which case, I would temporarily transfer the role elsewhere until the problem is taken care of on vDMC..

Yes... definitely do not snapshot domain controller, that could cause problems in your AD, as there would be out of sync data in your replication environment. In any case, I hope this won't be a problem in this case.

That is correct the INFRASTRUCTURE operations master was the machine that went down. It is up now but out of sync with the AD1 RID operations master.

oh ok , in this case, and if it's up, definitely do NOT seize the role.  
But, I would think a forced replication should bring everything back in sync.

I just looked at AD1 and when I open the setings for the Operations Masters to check it's setings I got something interesting.
On vDMC it shows have a RID Active Directory OPERATIONS MASTER called AD1
I have a PDC Active Directory OPERATIONS MASTER called AD1
I have a INFRASTRUCTURE OPERATIONS MASTER CALLED vDMC

But when I look at the same thing on AD1:
 RID Active Directory OPERATIONS MASTER called (AD1)
IPDC Active Directory OPERATIONS MASTER called (AD1)
INFRASTRUCTURE OPERATIONS MASTER CALLED  (ERROR)

 it shows an ERROR on the INFRASTRUCTURE machine.
The error is: OPERATIONS MASTER:  ERROR
The current operations master is offline.
The role cannot be transferred.

Well this could be the result of two things, I would think:
1- The fact that your Operations Master DC has crashed irrecoverably.
2- That your reverted to a snapshot of it, and now it doesn't know where it is in the environment.

I'm assuming you attempted to do a replication from AD1 to vMDC with no luck. right?

At this point, I'm thinking your options are as follows: (and this is assuming your vMDC hasn't been down for too long yet):
1- If you have a recent backup of the system state of vMDC, restore that. (i hope this will work as expected after reverting to a snapshot.
2- Seizing the operations master role to AD1, rebuild vMDC, re-move the Infrastructure Master back to it, and force a replication.

I'm assuming you attempted to do a replication from AD1 to vMDC with no luck. right?

No I have not done the replication. I was waiting for  a concensus that that is the right thing to try.
And (2)..how would I do that?

AD1 is up and working fine.

vDMC is running but out of sync with AD1.

Ok. well so far, the consensus is looking like it's me, and Mike popped in and popped out. But, what we're trying is not dangerous at this point, so there's no harm in attempting it.

If you have replmon, (it's in the support tools, that's simple way to do it).
Add your AD1 server, then right click on AD1 within replmon, and select "Synchronize Each Directory Partition with All Servers"
Then on the pop up screen, select : Disable Transitive, and Push Mode. If your vMDC is across different sites, then select the 3rd option as well: Cross-Site Boundaries.
Then select OK.

That should do it. Let's see if this will actually get things back in sync. after you do that, let's check your eventvwr, and see what errors we get.

The results in REPLMON say the replication failed because RPC server is unavailable.

Ok... well before you do anything drastic regarding the RPC. Usually, this error is often caused by faulty DNS configuration/replication
Can you run a Netdiag /v on both servers, and see what results you get? we need to make sure that your DNS is in good shape for this to work.

Sorry, during the workday I'm in and out

If you can't get replication to work option 2 above is what I'd go with

Step by step about halfway down here  http://blogs.technet.com/b/askds/archive/2009/06/05/dc-s-and-vm-s-avoiding-the-do-over.aspx  start at the section "...1) Forcefully demote the DC by running dcpromo /forceremoval...."

You will be force the demotion, metadata cleanup, seize the role and repromote.

Do you know what caused the blue screen?  I'd look into that too; don't want it to happen again on this box.

Thanks

Mike

The Blue Screen was caused by the installation of the CITRIX XenTools upgrade. I have them insatlled now and it seems to be running fine. Now just to clean up this mess.

As for the RPC error. It was : The Active DIrectory RPC Server could not be found.
I ran the replmon on the AD1 win 2003 server since there does not seem to be a 2008 ver of that tool available or installed for me at this time. I can download and install the 2003 ver if necessary on the vDMC win 2008 server if you want.

dcdiag /v will also give you info,  repadmin /showreps and /showrepl will be useful.  I'm guessing they haven't replicated since the incident happened.

The netdiag and the replmon do not run in 2008 server R2. I guess the tools are not out for it yet?
I didn't try to install the 2003 version of the tools yet. Should I?

Hmm. Yes, netdiag and replmon should work just fine in Windows 2008, you can go ahead and install them. Or if you don't want to install the whole support tools suite on your w2k8 server, just copy the netdiag and replmon utilities over. AFAIK, they don't have any dependencies.

I am not clear on the FMSO (Flexible Single Master Operations) roles.
The vDMC is also a DHCP and DNS server as well as a AD and Global Catalog server.
Does this matter?

Sorry on the typo FSMO (Flexible Single Master Operations) roles.

There are a couple of things to consider:
DHCP:
You definitely have to move over to a different server IF you want to actually REBUILD the server. If you want to just demote it and re-promote it, you don't need to do anything with the DHCP server.

Your DNS:
If AD1 is also running an AD integrated DNS zone, then it's not a big deal to temporarily remove your DNS server from vDMC.
However, remember, that your vDMC was also the DNS server to some of your clients. If your DHCP scope option was specifying the DNS server list, make sure you remove the IP of vDMC from the list, or your clients will start trying to connect to a non-existing DNS server.

Global Catalog:
Just enable AD1 to be a Global Catalog for now, until you straighten out the situation with vDMC.

So, action points
(assuming you want to rebuild completely)
1- Change your DNS client list in your DHCP server scope options to point only to AD1 for now.
2- Remove DNS from vDMC.
3- Remove vDMC from being a GC, and make AD1 a GC.
4- Try to demote vDMC (with dcpromo /forceremoval option).
If that doesn't work:
ntdutil metadata cleanup, and go through that process to clean up the environment as well as seize the Infrastructure FSMO role.

(Assuming you just want to demote and re-promote:
1- Keep your DHCP server on vDMC, demoting will not affect DHCP (so long as the server stays a member server at least).
2- You can optionally keep DNS running on a demoted server, personally, I would remove it while you're doing the work.
3- Remove vDMC from being a GC, and make AD1 a C
4- Try to demote vDMC (with dcpromo /forceremoval option), if that doesn't work, metadata cleanup and seize Infrastructure Role.

I would like to get the replication working without rebuilding the machine if possible.

I tried the dcpromo /forceremoval and I got a new error message:
Before you can  install or remove  Active Directory Domain Services, you must remove Active Directory Certificate Services. For information about the consequences of removing Active Directory Certificate Services, see Help and Support.

Ooh yikes.. vDMC is a Cert Authority?? Hmm.. that complicates things.
Well, in order to demote your DC, you need to first remove your Cert Authority from it. In order to do this, you definitely need to have it exist somewhere else in your environment, as it is probably being used by some services. Here are 2 documents that will help do so:
http://technet.microsoft.com/en-us/library/cc779540(WS.10).aspx 
http://technet.microsoft.com/en-us/library/cc755153(WS.10).aspx 

I haven't found anything quite specific to migrating your CA Authority to Windows 2008 Server, but it should work the same, A couple of steps may be a bit different due to the tools being relocated in 2008.

Once your Cert Authority is moved, you should be able to demote your DC.

Actually, the Certificate Authority is running on the 2008 Server. I'm thinking that moving the Infrastructure role and the Certificate Authority to AD2(Windows 2003 ENt Server) would be a pretty good idea. The only thing I'm not sure about is comments that I've read that state that I shouldn't have the Infrastructure FSMO running on a Domain Controller that is also a Global Catalog holder.
I only have ONE domain. Hummm

That should be fine for now. It's not going to break anything.
In a Best Practice Scenario, yes, it shouldn't be. Once you're back up and running you can separate the Infrastructure Role from the Global Catalog. For now, don't worry about it.

in one domain it really doesn't matter, in one domain just make every DC a GC, the Infrastructure master and GC roles don't really have a role in a single domain though.  

thanks

Mike

I am the NEW Systems Administrator at this site and I inherited this Network and half-baked CITRIX XenServer installation so I'm still trying to figure out what the previous sys admin was thinking.
It is my understanding that Mainly Smart Cards use the Certificate Authority to manage keys. We don't use smart cards here, so I'm thinking I can just un-install the Certificate Authority then do the dcpromo /forceremoval.  I do have Exchange Server 2007 running in my network too by the way.
Just so you know.

Actually your certificate authority can be active for any servers that are using a home-brew certificates ( i.e Local web servers with SSL certs, etc .. )

I wouldn't think Exchange would be an issue with the Cert Authority, as OWA would probably have a legit cert (Thawte, Verisign, Digicert), where the root cert is publicly available. (of course, unless the previous admin had no clue what they were doing, in which case, I feel for you ! :)

Is there a way I can see if the CA has isued any certs and to who or what?
That may help me decide if it's alright to KILL IT.
And I am starting to wonder myself if the previous sys admin had any real plan at all. :(
He left absolutley NO documentation on the network design or the CITRIX XenServer design (in progress) . I've been working two and a half months trying to get it disected the best I can.

*Dave*

You can open the certification authority and the certificate snap in MMCs...I'm not a cert expert but man this guy did leave you in a jam.

Do you sign or encrypt your emails? GK already talked about the SSL portion.

Thanks

Mike

Dave,

If that makes you feel any better. When I came to my current company, I inherited 350 Servers, about 25 child domains with external and internal trusts, a completely messed up AD, DNS, replication, and a horrible Citrix installation running my ERP system.  it took me 3 years to get rid of all the additional domains. I'm now down to 3 organized and documented domains, and a working AD infrastructure.

there was zero documentation from the previous person.  So believe me when I tell you, I feel your pain. It's a lot of work. but the upside is that you'll get a chance to rip things out and put them in production like you want them. Now it's up to you to document and establish things based on best practice :)

We're here to help... as much as we can/know at least :)

No, we don't encrypt our e-mails. I am looking at the CA ISSUED certificates node and it shows the certs issued. there is like 4 used by the vDMC itself. There are others though for some machines here. I'll have to check with the Netowork Admin to see he has any clues.
Yea, the previous admin got burned out & cracked , and quit. No transition.

I have asked around and it seems that my Certificate Authority has passed out some Certs, So I can't just kill it and forget it. So here is my plan for you to comment on.
1) backup the certificate authority using its own tool to a network drive.
2) remove the CA from that server
3) try to use a GUI to gracefully remove the server as a domain controller thereby automatically transferring any FSMO roles to some other DC in my network.
4) Promote the server back as a DC and GC holder
5) reinstall CA and perform a restore of the certs and keys.

Comments please

*Dave*

My DNS replication / WINS / CITRIX and AD are all undocumented and bairly functional. So I do look forward to doing it with a PLAN correctly, and documenting it, and keeping it going lean & mean.

I believe they have had 3 Sys Admins here in the last 5 years. None of them documented anything.

*Dave*

You could try the graceful transfer of the FSMO role prior to the demotion.

Aside from what Mike said, I think it sounds like a good plan.
I personally never had to transfer a CA authority, so please make sure that you do read any docs regarding this procedure, just in case there are some caveats that you need to be aware of.

I tried to transfer to Infrastructure role to my AD2 uising AD Userrs and COmputers. It shows the vDMC as offline in the AD server listing. Anyway, I connected to AD2, went to the Infrastructure tab and selected to transfer the role from vDMC to AD2 it complained that it could not connect to vDMC and asked if I wanted to do a forced transfer. I replied YES. So AD2 is an ACTIVE working AD server in my domain. It also complained that the AD2 has a Global Catalog on it too but it let me move it there. I suppose it is operating there now.
I tried to do a DCPROMO on vDMC to demote the machine and it won't let me because the machine is also a Certificate Authority. It instructs me to remove that role first then try it again. I backed up the CA files to a network drive, went to the Server Manager to remove the CA role and it did not work. It gave me the error:
ACTIVE DIRECTORY CERTIFICATE SERVICES
Attempt to un-install Certification Authority Web Enrollment Failed with error code 0x80070534. No mapping between account names and security IDs was done.

The following role servers were not removed CERTIFICATION AUTHORITY WEB ENROLLMENT.

So I am caught in a catch 22 here. I can't remove CA and until I do I can't demote the server.

*Dave*

My understanding of Certificate Authority is that you really can't transfer the certs. But you can do a backup of them, build a new server, give it the same name, install CA on it and restore the Certs to it. That's what they consider an Upgrade for the purposes of installing a new server to replace an end of life unit. So I do have the CA backed up. Once I get this straightend out I'll reinstall the server CA role and restore the database to it. That's the plan anyhow...

*Dave*

Another topic:
I know how to find out who has the RID / PDC / Infrastructure ROLES.
How can I determine who has the Schema Master and the Domain Naming Master Roles?

*Dave*

on a domain controller type this:
netdom query fsmo

Great! It worked wonderfully.

This was a tough one. Microsoft spent 5 hours in remote control rebuilding everything before it worked right.

wow ... after all my attempts to help, I got zero credit? kind of unfair...

Humm, I don't know how to SPLIT credit?

No problem on splitting, you can select request attention and tell the mod you want to split.
Glad you are up and running, looks like Microsoft earned their money on this one :)