SYSVOL corrupted

Hello,

Environment 2 DC, 60 PC clients, Windows 2008 R2 DC and Windows 2012 R2 DC

Last week we had a Cryptolocker and all files in network shares were corrupted
We restored all files from backups buy now we noticed that DC replication do not work any  more and NETLOGON script are not replicated (repadmin /replsummary has also errors)

Actually it appears that SYSVOL files have been corrupted on both DCs

Example of files found in the SYSVOL volume in POLICIES folder
GPT.INI.[mk.liukang@aol.com].wallet
admfiles.ini.[mk.liukang@aol.com].wallet
ect...

Is it possible to restore the SYSVOL volume from a Windows Backup on a USB drive ?
Is it possible to clean SYSVOL volume from  these corrupted files ? (we do not use GPOs)
What are my solutions to have replication work again with a clean SYSVOL?

Thank you very much
Gad SAADIAManagerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ChrisSr. Systems EngineerCommented:
Based on the info you posted here,
Yes, it is possible to restore AD and sysvol from a windows backup if you have a full file and system state backup of the DC.
You will need to isolate a DC, recover it, verify, and then force a replication after you've determined that its properly restored.

https://technet.microsoft.com/en-us/library/cc816596(v=ws.10).aspx

Once you have a DC back up and running, you'll need to force push to the other DC (Because they are most likely out of sync.)

By default this does a pull replication - which is how AD works by default. To do push replication use the following command:

repadmin /syncall /APeD

A = All Partitions
e = Enterprise (Cross Site)
D = Identify servers by distinguished name in messages.
P = Push

You want to do a push replication if you make changes on a DC and you want to replicate those changes to all other DC's.  For example, you make a change on DC1 and you want all other changes to get that change instantly, run repadmin /syncall /APeD on DC1.

For all repadmin syntax :
http://technet.microsoft.com/en-us/library/cc736571(v=ws.10).aspx

You could also initiate a replication from Sites and Services - https://technet.microsoft.com/en-us/library/cc816926(v=ws.10).aspx
Gad SAADIAManagerAuthor Commented:
Thank you very much for your help

Here more detailed information:

DC1: Physical server, Windows 2008 R2, 4 FSMO roles, complete server Windows Backup on USB drive
DC2: Hyper-V VM, Windows 2012 R2, complete VM VEEAM backup on NAS drive

All DATA folders and files have been restored OK for DC1 and DC2
presently both DCs are operational (with corrupted SYSVOL)

I would like to restore ONLY  AD and SYSVOL (not data files)
Should I restore on DC1 or on DC2? Any recommendation?

Is it dangerous to reboot server with corrupted SYSVOL? (servers has not been rebooted since infection)

Thank you very much
Shaun VermaakTechnical SpecialistCommented:
Just restore SYSVOL to an alternate location and replace on one of the DCs and check that it replicates to other DC. If ransomware created files with weird extensiond and shortcuts, delete them.

By default, even Domain Admins do not have write access to NETLOGON/SYSVOL via the share so encryption probably happened via admin shares.

I would strongly recommend you change all your user's passwords and review your domain admin group members
Amazon Web Services

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Gad SAADIAManagerAuthor Commented:
So if I understand well

1) I use Windows Backup to restore SYSTEM STATE to an alternative location (C:\TOTO for example)
2) I replace restored SYSVOL to damaged SYSVOL (replacing damaged SYSVOL). Should it be possible? How do I do that ? regular copy?
3) Replaced SYSVOL will replicate to other DC

It is OK?
Anyone has tested this solution?

Thank you
Shaun VermaakTechnical SpecialistCommented:
1) I use Windows Backup to restore SYSTEM STATE to an alternative location (C:\TOTO for example)
Yes
2) I replace restored SYSVOL to damaged SYSVOL (replacing damaged SYSVOL). Should it be possible? How do I do that ? regular copy?
Yes and you need to delete extra files created by ransomware
3) Replaced SYSVOL will replicate to other DC
Yes

It is OK?
Anyone has tested this solution?
SYSVOL is just a DFS. If it makes you feel more comfortable, backup first and copy new data in parts, such as only folder or policy foldet at a time
Gad SAADIAManagerAuthor Commented:
Hello,

I noticed that when I restore "full system state" through Windows Backup to an alternate location restored SYSVOL is still corrupted (randsomeware files appears in Sysvol) . Very strange. And when I restore from the same backup, same day only SYSVOL folder (like a regular C:\Windows\SYSVOL folder) it is not corrupted (no randsomware files appears)

Instead of copying SYSVOL folder from restored backup is there a way to just restore a functional SYSLOG from scratch (we do not have GPOs, scripts, or anything like that) so that replication  between the 2 DCS starts working again (with a new and functional SYSVOL) ?

THank you very much for your help
ChrisSr. Systems EngineerCommented:
If you're going to attempt to rebuild it from nothing, you should prepare yourselves for the possibility that you may need start over from scratch.. Just want to put that out there, so you begin to lay groundwork on expectations.

How to temporarily stabilize the domain SYSVOL tree

    Stop FRS on all domain controllers in the domain and set the service to Disabled.
    Manually copy the full set of policies to the following folder on each domain controller:
    \SYSVOL\SYSVOL\dns domain name\policies
    Typically, the following two policies are required for authentication:

        Default Domain Controllers Policy{6AC1786C-016F-11D2-945F-00C04fB984F9}
        Default Domain Policy {31B2F340-016D-11D2-945F-00C04FB984F9}

    Note You may have to copy additional policies depending on Group Policy requirements for the environment.
    Manually copy all necessary scripts to the following folder:

    \SYSVOL\SYSVOL\DNS Domain name\scripts

https://support.microsoft.com/en-ca/help/315457/how-to-rebuild-the-sysvol-tree-and-its-content-in-a-domain

If both are corrupted, and you cannot get FRS to replicate, and none of your restores appear to have valid data, then you may be out of options. The only other thing I can think of, is to take one DC offline, boot the other DC into Directory Services Restore Mode, and use BCDEdit to perform a restore to one DC. Then Seize all the FSMO roles. Run on that single DC, verify everything is "hopefully" working. Then, if you can finally get everything working, re-promote another DC back into the Domain.

- https://technet.microsoft.com/en-us/library/cc816897(v=ws.10).aspx
- https://technet.microsoft.com/en-us/library/cc794755(v=ws.10).aspx

Option - You could engage MS support. (They might have a trick or 2 up their sleeve) - With that said, that engagement will likely cost some money.
DrDave242Senior Support EngineerCommented:
(we do not have GPOs, scripts, or anything like that)

If you're really not using GPOs at all, the dcgpofix command can be used to recreate the two default GPOs (Default Domain Policy and Default Domain Controller Policy) from scratch with their default settings. Any modifications that were previously made to these GPOs will be lost, but from what you've said, you haven't made any modifications to them.

This command requires that the SYSVOL folder hierarchy be in place first. If it isn't, an error will let you know.
Gad SAADIAManagerAuthor Commented:
OK thank you

ANother question :

Can I just erase all files and folders from  SYSVOL  and run dcgpofix?
Windows will allow me to erase all SYSVOL content?

Steps:
- NET STOP NTFRS
- manually erase all SYSVOL files (not folders)
- run DCGPOFIX
- NET START NTFRS

And new SYSVOL will be replicated the other DC SYSVOL

Does that seems OK?

Thank you
DrDave242Senior Support EngineerCommented:
I believe this would work, but are you sure that SYSVOL is being replicated by FRS in your environment and not DFSR? FRS is the older, less efficient mechanism for replicating SYSVOL, but it may still be in use in your environment. Look through the FRS event log on at least of your DCs, and you should be able to tell pretty quickly whether it's in use. (You may also run across errors that indicate why SYSVOL isn't being replicated, a "journal wrap" condition being a very common cause.)
Gad SAADIAManagerAuthor Commented:
Yes SYSVOL is replicated by FRS

My problem with SYSVOL is that it has been infected by a Crypto VIrus.
ANd files insde SYSVOL are now corrupted (please see attached Word file)

Thank you for your help
DrDave242Senior Support EngineerCommented:
OK. Go ahead and proceed with those steps, as you've got nothing to lose. If SYSVOL still doesn't replicate correctly after you've done that, it may be necessary to perform an authoritative reinitialization of FRS on the same DC on which you run the dcgpofix command, and possibly a non-authoritative reinitialization on any other DCs.

The steps for doing this are here. Note that the value that you set for BurFlags determines whether an authoritative (0xD4) or non-authoritative (0xD2) reinitialization is performed. 0xD4 should only be used on the DC where you ran the dcgpofix command.

Are you confident that the ransomware has been eradicated, BTW?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.