SYSVOL & NETLOGON Corrupted after ransomware and original DC corrupted - Need Assitance with recovery order

I had this question after viewing SYSVOL corrupted.

I have a server that was fully corrupted by ransomware without a good restore option available.

I now know I need to rebuild the NETLOGON and SYSVOL shares from scratch and plan to do that per this article:
https://support.microsoft.com/en-us/help/290762/using-the-burflags-registry-key-to-reinitialize-file-replication-servi

I also know I need manually seize the roles and remove the old DC:
https://community.spiceworks.com/how_to/9942-complete-force-removal-of-a-domain-controller-from-active-directory-guide

The question I have is:

Which to do first?   I imagine I need to fix the shares first as they are required for proper AD operation, though I fear that will fail due to the lingering DC.  

Perhaps someone here has done this before?

Thanks in Advance,
Fred
LVL 2
fredimacAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Cliff GaliherCommented:
You can't simply repair the shares. If something was able to get write access to the domain controller, which is the only way ransomware could corrupt those shares, and you had no pre-compromised backup, the domain is toast. You are rebuilding the domain from scratch, not just repairing the shares.
fredimacAuthor Commented:
As far as I can tell, the only thing that was affected was the NETLOGON and SYSVOL files.

Can you be a bit more specific about why you think the whole domain is toast?  

Is it just a loop I am in that I can't fix a missing DC with broken shares and I can't fix the broken FRS shares with the missing DC?  

Thanks,
Fred
Cliff GaliherCommented:
Sysvol is how DCs replicate. So corruption in sysvol basically is a corrupted domain.

But more importantly, the only way to corrupt them is to have write access on a domain controller. Which means other things could also have been written...and replicated.

There is a reason why domain controllers are considered high priority for protection, are high priority targets, and why domain admin credentials should be given sparingly, used even more sparingly, and why so much effort is out into hardening those systems.

And why backups are important.

None of that seems to have happened here, and yes that means rebuilding.

It is kind of like slamming a car into a tree at 120mph and then asking how to fix a scratch on the rear passenger door.  The car is still totalled.
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

fredimacAuthor Commented:
Also, to be clear, an old DC had it's files encrypted.  A new DC received SYSVOL and NETLOGON files during replication.  There are no other encrypted files anywhere on the new DC other than SYSVOL and NETLOGON.
Cliff GaliherCommented:
But sysvol is where the entire AD database lives. So when you say they "only are corrupt, that is still a corruption of everything that makes a DC what it is. So...yeah, your domain is toast.
fredimacAuthor Commented:
My question still stands then:  

Should I attempt an authoritative restore via the BurFlags method before manually removing the old DC or the other way around?
Cliff GaliherCommented:
Neither.  Burflags are used to address a very specific scenario where replication is broken. What you described is not broken replication. The data got corrupted, but that corrupted data replicated just fine. So the order you do things won't make a difference. Without a backup, you have no good copies of the data to restore.
fredimacAuthor Commented:
Please review the contents of this request.   It seems to have worked for this person:
https://www.experts-exchange.com/questions/29002511/SYSVOL-corrupted.html
arnoldCommented:
remove the old compromised domain controller from the network. seize (ntdsutils) the roles and go through the process of setting up sysvol/netlogon.
Note you either do it right now, or go through this process as you intend and repeat this process down the line when as Cliff on three separate occasions pointed out that your Entire AD might be completely compromised i.e. the DC you think is not compromised might be, but has not shown any manifestation of that issue yet.
fredimacAuthor Commented:
I suppose the alternative is to remove AD and reinstall, re-add users and un-join / re-join all workstations to the domain.

Going forward there will be only one DC and they use almost no GPO's so ...  it might be worth it to try the fix first, then do the full Domain rebuild if need be.   I really don't think the ransomware is on the DC.  It was clearly on the old server and hit others just via Shares.

In the case of this client, the backup repository was encrypted as well do to a forgotten mapped drive.
arnoldCommented:
sysvol/netlogon hit means the account compromised by ransomware was administrative.
Hey, its your decision and your time spent.

if you've not backed up the documents from the shares, as of yet, you should do that first thing and make sure that data does not contain the virus/worm/compromised download.............

another post here had the use of csvde to export the user accounts ...........
Cliff GaliherCommented:
" I really don't think the ransomware is on the DC.  It was clearly on the old server and hit others just via Shares."

It was on a machine with DOMAIN ADMIN credentials. It could not have corrupted the shares on other machines without them. Once it had the credentials, it could inject code whenever and however it wanted.

But that is all still besides the point. The procedure you want to do simply doesn't apply to your situation. To use another analogy, you have a broken leg and are asking the doctor to perform heart surgery to fix it. That just isn't how things work.

You've now has a second expert confirm what I've suggested. Why ask experts for help of you want to argue and ignore their advice?  It isn't my infrastructure so...frankly...do as you please. It doesn't bother me any which way, except for the time i feel I wasted trying to help. But think long and hard why you came here in the first place. Because it doesn't seem like you've thought that through.
Shaun VermaakTechnical SpecialistCommented:
If you have any data available for the SYSVOL you might be able to recover using this process
https://www.experts-exchange.com/articles/29522/How-to-restore-Active-Directory-Group-Policy-with-only-SYSVOL-data.html
arnoldCommented:
Cliff, I think you reversed your analogy. I.e. person diagnosed with a serious condition, but wants a remedy of a mundane condition.

 As you noted, once an administrative account is compromised, the reliability of the workstation existing condition might also be in doubt...

at times people seek to have their approach reaffirmed while knowing full well what the right approach is.
fredimacAuthor Commented:
OK, OK, I get it ;)  The best thing to do is to be sure nothing carries over.   I will recreate the Domain.  

I am still curious about the user in this case who said the "restore from scratch" option worked after all data encrypted.  https://www.experts-exchange.com/questions/29002511/SYSVOL-corrupted.html

I've sent them an inquiry.

Thanks,
Fred
Cliff GaliherCommented:
There is a fundamental difference between your situation and theirs. Right in their question, they said they restored from a backup. So they ONLY had to fix replication.

From your question:  "without a good restore option available."

If you had a known good backup, my advice would be different.  But by your own admission, you don't. And that is also why a replication repair option was possible in their case and not in yours.
LearnctxEngineerCommented:
Sysvol is how DCs replicate. So corruption in sysvol basically is a corrupted domain.

Cliff, I'm going to have to correct you here. Otherwise some people might be left with a gross misunderstanding of the purpose of the Active Directory Database and SYSVOL.

The Active Directory database does not use SYSVOL to replicate; it never has and it never will. The Active Directory database is not stored in SYSVOL either. The Active Directory database and transaction files are stored (by default) under C:\Windows\NTDS, SYSVOL is stored (by default) under C:\Windows\SYSVOL. SYSVOL has its own set of replication technologies (FRS or DFS-R). SYSVOL stores your group policy files and your logon script files (via the NETLOGON share). Active Directory replication is far more complicated than FRS and DFSR could handle.

If anyone would like a basic refresher course into Active Directory see part 1 and part 2. If you would prefer a deep dive into AD replication, see here.

I would agree though with what others have said; if you had a domain controller which had SYSVOL compromised then you need to seriously consider that the DC has been completely compromised. If you are not 100% certain, then assume it has been. The normal course here is to:

  1. Engage Microsoft AD PFE's
  2. Engage a security consultant to perform a forensics analysis to figure out what has been done (MS or other company)
  3. Fix what has been broken or start from scratch

Basically follow the 3 R's of a good administrator Recover, Restore, Resign. Someone definitely needs an ass whooping for not having backups if you don't have them.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
fredimacAuthor Commented:
Thanks all for your comments and thanks for the clarification @Learnctx!

FYI, a forgotten mapped drive to the backup repository containing all the server image backups, which had been current, was encrypted along with the server.  So backups were good.  Mapped drives are BAD.  Many steps have been taken to secure the backup target and add redundancy to the backups since this happened.  So yes, self flagellation continues ;)  

Believe it or not, I actually did want an answer to my original question.  Perhaps I should have opened it with "I know the best solution is... but I still want to know, given my situation".   The easy "answer" is to plow everything under.  For now, I will settle for backups that update versions every three hours, onsite and offsite.

The question was:   Should I remove the old DC first, or "fix" the SYSVOL problems first.

The DC is functional enough at the moment.  I would appreciate some pointers from someone experienced in the black art of repairing the SYSVOL.   I have removed tomb-stoned DC's enough times that I am comfortable with it, but I wonder if the SYSVOL should be fixed first.
arnoldCommented:
what do you mean repair. you need the default domain controller and default domain policy to regenerate?
the GPO's should exist in the AD and copied out to the SYSVOL.
https://social.technet.microsoft.com/Forums/windows/en-US/95daaa73-5916-4711-8c57-fa130fe87418/gpotoolexe-in-windows-2012-r2?forum=winserver8gen
https://technet.microsoft.com/en-us/library/hh875588(v=ws.11).aspx

Use of terminology makes it unclear what you mean?
The data encrypted, the structure remains. i.e. the shares are available.
Cliff GaliherCommented:
If you keep asking the same question, you'll keep getting the same answer. The order doesn't matter because you can't repair your shares.  It isn't that I am trying to push you to a different solution. It is that you are asking how to do something that can't be done.  Ride a bicycle into a mechanic's shop and ask him how to "fix" the engine....

And yes, now you'll bring up that it "worked" for another question.  BUT.  Same question, same answer.

That question *STARTED* with them restoring a backup. You have no backup. So your situations are not an apples to apples comparison. They had an apple. You have a duck.  

Without a backup, and without a decryption key, and without a healthy DC, what was in those shares is gone. Not repairable. GONE.  Setting a burflags reg key won't suddenly bring the data back, whether you do it before, after, during, sacrificing a chicken, standing on your head, submerged in tar, while chanting the Russian alphabet backwards....GONE.  I don't think I can be more blunt.
PberSolutions ArchitectCommented:
No comment has been added to this question in more than 21 days, so it is now classified as abandoned.

I have recommended this question be closed as follows:

Split:
-- Learnctx (https:#a42272102)
-- Cliff Galiher (https:#a42273510)


If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.

Pber
Experts-Exchange Cleanup Volunteer
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Ransomware

From novice to tech pro — start learning today.