Active Directory repair after roll back. Hyper-V virtual machine

Aaron Moukperian
Aaron Moukperian used Ask the Experts™
on
Recently rolled back our AD server (Windows Server 2008) due to a ransomware issue. After doing an instant recovery (Veeam backup & Replication) of the VM, the active directory is now corrupt. Is there a way to repair it?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Daryl BamforthTechnical Expert

Commented:
Not really, AD does not work well with snapshots,

Have a look at this Microsoft Article for considerations when running DC's in Hyper-V.

https://technet.microsoft.com/en-us/library/virtual_active_directory_domain_controller_virtualization_hyperv(WS.10).aspx

Specifically the back-up sections.

Do you have any other back-ups?
How big is your environment? (How many users/objects?)
MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017

Commented:
The only recommended way is recover backup.
Do you have another backup?
Do you have  ADC?

Author

Commented:
I only have the other roll back dates within Veeam, no other backup in place.  We only have the one DC running AD, there is another VM running the exchange server. It's not a large environment only 40 AD users
Success in ‘20 With a Profitable Pricing Strategy

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017

Commented:
You can try with another date backup restore from Veeam but not 100% sure it will work.
Daryl BamforthTechnical Expert

Commented:
Probably quicker to start from scratch.

Look though the microsoft link,  especially at the backups sections.
MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017

Commented:
What is the error u get when u try to login after restore backup?
Can you see "Applying computer settings"?

Author

Commented:
I can get into the server but when I go into active directory and I access a user when I try and to view the member of tab i get operation error has occurred. Also, users are unable to login to the domain.
Daryl BamforthTechnical Expert

Commented:
Have the AD services been started back up?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
do you only have a single DC ?

Author

Commented:
The Active Directory Domain services and web services both show started

Author

Commented:
There is only the single DC

Author

Commented:
I also get an error on the Exchange VM when trying to access the exchange management console "Kerberos" authentication failed: connecting to the remote server failed with the following error message: The WinRM client received an HTTP server error status (500) the remote service did not include any other information about the cause of failure. This started after the rollback.
Daryl BamforthTechnical Expert

Commented:
The Exchange error is because Exchange was linked into your Active Directory. With this gone it cannot authenticate any more.

Author

Commented:
I can log them in with the administrator account on the domain just not their own login if that matters
Daryl BamforthTechnical Expert

Commented:
which Administrator account? The Domain Admin one? Or machine local?

Author

Commented:
The domain admin is working on their machines
Daryl BamforthTechnical Expert

Commented:
Can you get into the AD management console?

Author

Commented:
I can access the AD console and I can see all the users
Daryl BamforthTechnical Expert

Commented:
What is the date/time of the AD? if you rolled it back is it wrong?

Try re-adding one of your PCs back into the domain (reset the pc object), reset a users password and see if they can now log in.

Restoring from backup may have broken the trust relationship between computer objects and the domain.

Author

Commented:
I checked the date and time it is correct on all servers. I'm attempted to reconnect a PC now

Author

Commented:
Looks like it can join the domain without error but when I go to log in, says incorrect password. Attempting to reset the password from AD gives An internal error has occurred

Author

Commented:
Is it not possible to repair AD using NTDSUTIL in active directory repair mode?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
do you not have any backups that you can restore which work ?

How many user and computers ?

Author

Commented:
I have other rollback dates from earlier in the week. Should I try one of those? I used the one from Thursday (the issue occurred Friday)
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
I would certainly consider using another backup. If none of them work, then you need to think about the future, as to why you backup if you cannot restore a backup correctly!

Also consider based on how many users and computers you have, the effort required, to just install a new DC , re-create all the user accounts, groups and computers based on how complex your AD is.
Lee W, MVPTechnology and Business Process Advisor
Most Valuable Expert 2013
Commented:
You should be performing test restores occasionally.  This would prove to yourself your backups work and help you learn what to do and what not to do when you have an actual emergency like this.

Definitely try previous backups.  It sounds like your backup software leaves the server in a state as if it was powered off unexpectedly.  It's definitely possible this backup is corrupt from an AD standpoint.  Earlier ones may well be fine.
MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017

Commented:
Agree with Lee.
You should test your backup, And Veeam allows to restore backup without effecting the production network.
Please try to restore another backup and update the status here.

Meanwhile create 2 new VMs and start installation of new domain controller and new exchange server.
Start copying the ED file from Exchange server to this server for mounting the DB.
In case if that restore fails you can continue creating these accounts and install Exchange and mount the database.
yo_beeDirector of Information Technology
Commented:
I think you maybe SOL and will have to rebuild your AD (users, groups, and computers) since you do not have a true backup using Windows backup or third party backup utility that uses agents to get file level backup.
Snapshot rollbacks are not supports for DC's as others pointed out.


40 users is not a mountain of objects that needs to be recreated.

You maybe able to to use powershell to export the users to a CSV and import them into the rebuild DC. If this is possible it will help speed up the rebuild process.

It's a little to late, but now you know the importance of a system state backup of your DC. Even with a small environment a second DC would be a big help for recovery and probably should be consider moving forward.

Author

Commented:
I tried running a repair on the database but to no luck. It's a small environment so I'll end up rebuilding it. I wasn't the one that setup this environment to begin with but I appreciate the tips.
yo_beeDirector of Information Technology

Commented:
At least you learned something and hope if your AD is compromised in the future you have a solid DR plan.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
1. Build two DCs.
2. Backup
3. Regularly Test your Backups.

otherwise don't bother backing up!

Author

Commented:
So since my current AD is corrupt I can't join a secondary DC to the existing. Is my only option to backup the data from the main DC format and reinstall?
Daryl BamforthTechnical Expert
Commented:
I think Andrew was on about for your new infrastructure.

Your absolutely right. A second DC at this point won't work due to corruption. If the main is a VM, just leave it for now so you have it as a point of reference for group membership etc.

Just out of interest ... did you look at this section in the link I posted above?

To restore the system state backup of a virtual domain controller
Start the domain controller’s virtual machine, and press F5 to access the Windows Boot Manager screen. If you are required to enter connection credentials, immediately click the Pause button on the virtual machine so that it does not continue starting. Then, enter your connection credentials, and click the Play button on the virtual machine. Click inside the virtual machine window, and then press F5.
If you do not see the Windows Boot Manager screen and the domain controller begins to start in normal mode, turn off the virtual machine to prevent it from completing startup. Repeat this step as many times as necessary until you are able to access the Windows Boot Manager screen. You cannot access DSRM from the Windows Error Recovery menu. Therefore, turn off the virtual machine and try again if the Windows Error Recovery menu appears.
In the Windows Boot Manager screen, press F8 to access advanced boot options.
In the Advanced Boot Options screen, select Directory Services Restore Mode, and then press ENTER. This starts the domain controller in DSRM.
Use the appropriate restore method for the tool that you used to create the system state backup. If you used Windows Server Backup, see Performing a Nonauthoritative Restore of AD DS (http://go.microsoft.com/fwlink/?LinkID=132637).

The important bit to not allow it to start up in normal mode when restoring, need to boot into DSRM mode first.
yo_beeDirector of Information Technology

Commented:
The secondary DC would have only been a viable if it was already joined prior to the corruption.

The damage has been done and you do not have a true backup of the system state so there is no way to do restore (non authoritative restore).

Author

Commented:
That's correct I don't have the Windows system backup. So I need to take the current corrupted Active Directory VM offline and bring up a new VM and start fresh?
Daryl BamforthTechnical Expert

Commented:
Yes, that's right.

Just disconnect it's NIC so you can still have the console open for it when you are creating users/groups/devices etc...
yo_beeDirector of Information Technology

Commented:
And rejoin all your workstations and servers.

Author

Commented:
Since the domain is on that server do I need to change the domain on the old server from or I simply disabling the nic will work?
MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017

Commented:
If you have an additional DC (ADC) you could seize the existing DC and make your systems up.
You should have working backup offsite. Test the backup and copy that backup to a NAS or HDD and save/keep it offsite on a monthly basis.
If you have budget please try to have a DR site.

If I am not mistaken there is not primary DC and secondary DC concept after 2008 server. It is replaced with DC and ADC.
FSMO holder is considered as DC. Please correct me if I am wrong.
yo_beeDirector of Information Technology

Commented:
You are technically creating a completely new domain.
Your old dc should not even exist anymore.
Daryl BamforthTechnical Expert

Commented:
As far as your new install is concerned it is the Primary DC as the domain only exists on the old, corrupted VM. So as long as that one is off the network (disconnected NIC does this nicely) the new VM will know nothing about it and be quite happy to install away and be authoritative for your domain.
MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017

Commented:

Author

Commented:
Thanks, I'm setting the new up the new VM now
yo_beeDirector of Information Technology

Commented:
You have some work ahead of you.  Good luck and stay calm.
yo_beeDirector of Information Technology

Commented:
Also remember to use a backup tool that is a true system backup and test it regularly.
MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017

Commented:
BTW If you do not have your Exchange database(EDB file) please export emails to PST on all your outlooks.
Once PC is disjoined from old domain you lost access to the previous outlook profile.
http://support.sherweb.com/Faqs/Show/how-to-export-an-outlook-2010-pst-file-exchange-2010
https://support.office.com/en-us/article/Export-or-backup-email-contacts-and-calendar-to-an-Outlook-pst-file-14252b52-3075-4e9b-be4e-ff9ef1068f91

Author

Commented:
Thanks everyone, for your input on this issue. I've since setup the new VM server with AD. Question regarding the old VM sever with AD, it is possible to just remove AD and have it act as a file server or will that potentially mess with the new AD Server?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
You could try and remove AD role which will uninstall all the AD components and return this server to a server in a workgroup and then you can add it to your new domain
yo_beeDirector of Information Technology

Commented:
Should be fine as long as the demotion is clean. You will need to rejoin the machine as a member server.

There will still need to apply new ACL since the AD is new.

Also best practice is not to have your file server live on your DC.

Author

Commented:
The last issue I'm having is being able to reconnect to the exchange server since the corrupt Active Directory. We're moving to office 365 but need to access the exchange server to migrate the old email. Do I need to connect the exchange server to the new AD?
MASEE Solution Guide - Technical Dept Head
Most Valuable Expert 2017
Commented:
Your Exchange server is broken as your Exchange integrated AD/domain is broken.
You have to setup a new Exchange in the new domain and mount the  DB.
or
You can export and import all emails to Office 365. There are 3rd party
This is xchange-to-office365-migrator.exe from  SocialTechnet gallery
https://gallery.technet.microsoft.com/EDB-to-Office-365-Software-4d1cf182 
https://www.systoolsgroup.com/exchange-to-office365-migrator.html
https://www.nucleustechnologies.com/exchange-edb-to-office-365-migration.html
Dariusz TykaICT Infrastructure Specialist Senior

Commented:
You mentioned that have instant recovered  domain controller using Veeam. So it looks like you have valid backups of domain controller. If your backup is properly configured (application aware processing was enabled within dc backup job settings) you should be able to do authoritative restore of this domain controller. So if you have backup prior to date when rollback occure it should work for you. Pls take a look here on how to do authoritative restore of dc:
https://www.veeam.com/kb2119
Unfortunately I'm not exchange expert so others needs to point you what to do with exchange server after authoritave restore of dc. And/or if it this is supported in such configuration. So it should be possible depending on jour job config.
PberSolutions Architect

Commented:
No comment has been added to this question in more than 21 days, so it is now classified as abandoned.

I have recommended this question be closed as follows:

Split:
-- Andrew Hancock (VMware vExpert / EE MVE^2) (https:#a42347055)
-- MAS (https:#a42347426)
-- Daryl Bamforth (https:#a42347198)
-- Andrew Hancock (VMware vExpert / EE MVE^2) (https:#a42347127)
-- yo_bee (https:#a42347118)
-- Lee W MVP (https:#a42347059)


If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.

Pber
Experts-Exchange Cleanup Volunteer

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial