Active Directory repair after roll back. Hyper-V virtual machine

Recently rolled back our AD server (Windows Server 2008) due to a ransomware issue. After doing an instant recovery (Veeam backup & Replication) of the VM, the active directory is now corrupt. Is there a way to repair it?
Aaron MoukperianAsked:
Who is Participating?
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would certainly consider using another backup. If none of them work, then you need to think about the future, as to why you backup if you cannot restore a backup correctly!

Also consider based on how many users and computers you have, the effort required, to just install a new DC , re-create all the user accounts, groups and computers based on how complex your AD is.
1
 
Daryl BamforthTechnical ExpertCommented:
Not really, AD does not work well with snapshots,

Have a look at this Microsoft Article for considerations when running DC's in Hyper-V.

https://technet.microsoft.com/en-us/library/virtual_active_directory_domain_controller_virtualization_hyperv(WS.10).aspx

Specifically the back-up sections.

Do you have any other back-ups?
How big is your environment? (How many users/objects?)
1
 
MAS (MVE)Technical Department HeadCommented:
The only recommended way is recover backup.
Do you have another backup?
Do you have  ADC?
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
Aaron MoukperianAuthor Commented:
I only have the other roll back dates within Veeam, no other backup in place.  We only have the one DC running AD, there is another VM running the exchange server. It's not a large environment only 40 AD users
0
 
MAS (MVE)Technical Department HeadCommented:
You can try with another date backup restore from Veeam but not 100% sure it will work.
0
 
Daryl BamforthTechnical ExpertCommented:
Probably quicker to start from scratch.

Look though the microsoft link,  especially at the backups sections.
0
 
MAS (MVE)Technical Department HeadCommented:
What is the error u get when u try to login after restore backup?
Can you see "Applying computer settings"?
0
 
Aaron MoukperianAuthor Commented:
I can get into the server but when I go into active directory and I access a user when I try and to view the member of tab i get operation error has occurred. Also, users are unable to login to the domain.
0
 
Daryl BamforthTechnical ExpertCommented:
Have the AD services been started back up?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you only have a single DC ?
0
 
Aaron MoukperianAuthor Commented:
The Active Directory Domain services and web services both show started
0
 
Aaron MoukperianAuthor Commented:
There is only the single DC
0
 
Aaron MoukperianAuthor Commented:
I also get an error on the Exchange VM when trying to access the exchange management console "Kerberos" authentication failed: connecting to the remote server failed with the following error message: The WinRM client received an HTTP server error status (500) the remote service did not include any other information about the cause of failure. This started after the rollback.
0
 
Daryl BamforthTechnical ExpertCommented:
The Exchange error is because Exchange was linked into your Active Directory. With this gone it cannot authenticate any more.
0
 
Aaron MoukperianAuthor Commented:
I can log them in with the administrator account on the domain just not their own login if that matters
0
 
Daryl BamforthTechnical ExpertCommented:
which Administrator account? The Domain Admin one? Or machine local?
0
 
Aaron MoukperianAuthor Commented:
The domain admin is working on their machines
0
 
Daryl BamforthTechnical ExpertCommented:
Can you get into the AD management console?
0
 
Aaron MoukperianAuthor Commented:
I can access the AD console and I can see all the users
0
 
Daryl BamforthTechnical ExpertCommented:
What is the date/time of the AD? if you rolled it back is it wrong?

Try re-adding one of your PCs back into the domain (reset the pc object), reset a users password and see if they can now log in.

Restoring from backup may have broken the trust relationship between computer objects and the domain.
1
 
Aaron MoukperianAuthor Commented:
I checked the date and time it is correct on all servers. I'm attempted to reconnect a PC now
0
 
Aaron MoukperianAuthor Commented:
Looks like it can join the domain without error but when I go to log in, says incorrect password. Attempting to reset the password from AD gives An internal error has occurred
0
 
Aaron MoukperianAuthor Commented:
Is it not possible to repair AD using NTDSUTIL in active directory repair mode?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you not have any backups that you can restore which work ?

How many user and computers ?
0
 
Aaron MoukperianAuthor Commented:
I have other rollback dates from earlier in the week. Should I try one of those? I used the one from Thursday (the issue occurred Friday)
0
 
Lee W, MVPTechnology and Business Process AdvisorCommented:
You should be performing test restores occasionally.  This would prove to yourself your backups work and help you learn what to do and what not to do when you have an actual emergency like this.

Definitely try previous backups.  It sounds like your backup software leaves the server in a state as if it was powered off unexpectedly.  It's definitely possible this backup is corrupt from an AD standpoint.  Earlier ones may well be fine.
0
 
MAS (MVE)Technical Department HeadCommented:
Agree with Lee.
You should test your backup, And Veeam allows to restore backup without effecting the production network.
Please try to restore another backup and update the status here.

Meanwhile create 2 new VMs and start installation of new domain controller and new exchange server.
Start copying the ED file from Exchange server to this server for mounting the DB.
In case if that restore fails you can continue creating these accounts and install Exchange and mount the database.
0
 
yo_beeDirector of Information TechnologyCommented:
I think you maybe SOL and will have to rebuild your AD (users, groups, and computers) since you do not have a true backup using Windows backup or third party backup utility that uses agents to get file level backup.
Snapshot rollbacks are not supports for DC's as others pointed out.


40 users is not a mountain of objects that needs to be recreated.

You maybe able to to use powershell to export the users to a CSV and import them into the rebuild DC. If this is possible it will help speed up the rebuild process.

It's a little to late, but now you know the importance of a system state backup of your DC. Even with a small environment a second DC would be a big help for recovery and probably should be consider moving forward.
0
 
Aaron MoukperianAuthor Commented:
I tried running a repair on the database but to no luck. It's a small environment so I'll end up rebuilding it. I wasn't the one that setup this environment to begin with but I appreciate the tips.
0
 
yo_beeDirector of Information TechnologyCommented:
At least you learned something and hope if your AD is compromised in the future you have a solid DR plan.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
1. Build two DCs.
2. Backup
3. Regularly Test your Backups.

otherwise don't bother backing up!
0
 
Aaron MoukperianAuthor Commented:
So since my current AD is corrupt I can't join a secondary DC to the existing. Is my only option to backup the data from the main DC format and reinstall?
0
 
Daryl BamforthTechnical ExpertCommented:
I think Andrew was on about for your new infrastructure.

Your absolutely right. A second DC at this point won't work due to corruption. If the main is a VM, just leave it for now so you have it as a point of reference for group membership etc.

Just out of interest ... did you look at this section in the link I posted above?

To restore the system state backup of a virtual domain controller
Start the domain controller’s virtual machine, and press F5 to access the Windows Boot Manager screen. If you are required to enter connection credentials, immediately click the Pause button on the virtual machine so that it does not continue starting. Then, enter your connection credentials, and click the Play button on the virtual machine. Click inside the virtual machine window, and then press F5.
If you do not see the Windows Boot Manager screen and the domain controller begins to start in normal mode, turn off the virtual machine to prevent it from completing startup. Repeat this step as many times as necessary until you are able to access the Windows Boot Manager screen. You cannot access DSRM from the Windows Error Recovery menu. Therefore, turn off the virtual machine and try again if the Windows Error Recovery menu appears.
In the Windows Boot Manager screen, press F8 to access advanced boot options.
In the Advanced Boot Options screen, select Directory Services Restore Mode, and then press ENTER. This starts the domain controller in DSRM.
Use the appropriate restore method for the tool that you used to create the system state backup. If you used Windows Server Backup, see Performing a Nonauthoritative Restore of AD DS (http://go.microsoft.com/fwlink/?LinkID=132637).

The important bit to not allow it to start up in normal mode when restoring, need to boot into DSRM mode first.
0
 
yo_beeDirector of Information TechnologyCommented:
The secondary DC would have only been a viable if it was already joined prior to the corruption.

The damage has been done and you do not have a true backup of the system state so there is no way to do restore (non authoritative restore).
0
 
Aaron MoukperianAuthor Commented:
That's correct I don't have the Windows system backup. So I need to take the current corrupted Active Directory VM offline and bring up a new VM and start fresh?
0
 
Daryl BamforthTechnical ExpertCommented:
Yes, that's right.

Just disconnect it's NIC so you can still have the console open for it when you are creating users/groups/devices etc...
0
 
yo_beeDirector of Information TechnologyCommented:
And rejoin all your workstations and servers.
0
 
Aaron MoukperianAuthor Commented:
Since the domain is on that server do I need to change the domain on the old server from or I simply disabling the nic will work?
0
 
MAS (MVE)Technical Department HeadCommented:
If you have an additional DC (ADC) you could seize the existing DC and make your systems up.
You should have working backup offsite. Test the backup and copy that backup to a NAS or HDD and save/keep it offsite on a monthly basis.
If you have budget please try to have a DR site.

If I am not mistaken there is not primary DC and secondary DC concept after 2008 server. It is replaced with DC and ADC.
FSMO holder is considered as DC. Please correct me if I am wrong.
0
 
yo_beeDirector of Information TechnologyCommented:
You are technically creating a completely new domain.
Your old dc should not even exist anymore.
0
 
Daryl BamforthTechnical ExpertCommented:
As far as your new install is concerned it is the Primary DC as the domain only exists on the old, corrupted VM. So as long as that one is off the network (disconnected NIC does this nicely) the new VM will know nothing about it and be quite happy to install away and be authoritative for your domain.
0
 
Aaron MoukperianAuthor Commented:
Thanks, I'm setting the new up the new VM now
1
 
yo_beeDirector of Information TechnologyCommented:
You have some work ahead of you.  Good luck and stay calm.
1
 
yo_beeDirector of Information TechnologyCommented:
Also remember to use a backup tool that is a true system backup and test it regularly.
0
 
MAS (MVE)Technical Department HeadCommented:
BTW If you do not have your Exchange database(EDB file) please export emails to PST on all your outlooks.
Once PC is disjoined from old domain you lost access to the previous outlook profile.
http://support.sherweb.com/Faqs/Show/how-to-export-an-outlook-2010-pst-file-exchange-2010
https://support.office.com/en-us/article/Export-or-backup-email-contacts-and-calendar-to-an-Outlook-pst-file-14252b52-3075-4e9b-be4e-ff9ef1068f91
0
 
Aaron MoukperianAuthor Commented:
Thanks everyone, for your input on this issue. I've since setup the new VM server with AD. Question regarding the old VM sever with AD, it is possible to just remove AD and have it act as a file server or will that potentially mess with the new AD Server?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You could try and remove AD role which will uninstall all the AD components and return this server to a server in a workgroup and then you can add it to your new domain
0
 
yo_beeDirector of Information TechnologyCommented:
Should be fine as long as the demotion is clean. You will need to rejoin the machine as a member server.

There will still need to apply new ACL since the AD is new.

Also best practice is not to have your file server live on your DC.
0
 
Aaron MoukperianAuthor Commented:
The last issue I'm having is being able to reconnect to the exchange server since the corrupt Active Directory. We're moving to office 365 but need to access the exchange server to migrate the old email. Do I need to connect the exchange server to the new AD?
0
 
MAS (MVE)Technical Department HeadCommented:
Your Exchange server is broken as your Exchange integrated AD/domain is broken.
You have to setup a new Exchange in the new domain and mount the  DB.
or
You can export and import all emails to Office 365. There are 3rd party
This is xchange-to-office365-migrator.exe from  SocialTechnet gallery
https://gallery.technet.microsoft.com/EDB-to-Office-365-Software-4d1cf182 
https://www.systoolsgroup.com/exchange-to-office365-migrator.html
https://www.nucleustechnologies.com/exchange-edb-to-office-365-migration.html
0
 
Dariusz TykaICT Infrastructure Specialist Senior Commented:
You mentioned that have instant recovered  domain controller using Veeam. So it looks like you have valid backups of domain controller. If your backup is properly configured (application aware processing was enabled within dc backup job settings) you should be able to do authoritative restore of this domain controller. So if you have backup prior to date when rollback occure it should work for you. Pls take a look here on how to do authoritative restore of dc:
https://www.veeam.com/kb2119
Unfortunately I'm not exchange expert so others needs to point you what to do with exchange server after authoritave restore of dc. And/or if it this is supported in such configuration. So it should be possible depending on jour job config.
0
 
PberSolutions ArchitectCommented:
No comment has been added to this question in more than 21 days, so it is now classified as abandoned.

I have recommended this question be closed as follows:

Split:
-- Andrew Hancock (VMware vExpert / EE MVE^2) (https:#a42347055)
-- MAS (https:#a42347426)
-- Daryl Bamforth (https:#a42347198)
-- Andrew Hancock (VMware vExpert / EE MVE^2) (https:#a42347127)
-- yo_bee (https:#a42347118)
-- Lee W MVP (https:#a42347059)


If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.

Pber
Experts-Exchange Cleanup Volunteer
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.