Avatar of mkavinsky
mkavinsky
 asked on

Unable to boot up Windows 2016 Domain Controller (physical server) due to NTDS (0xc00002e2)

Experts, 


really stumped on this one.  I have a physical server (Dell T330) with RAID1 (2 7.5TB Hard drives).  Server is a Windows Server 2016 standard, Domain controller.

there was a power outage the other day (was plugged into a battery backup) and the server went down.   Its not booting now and I get the error code: 0xc00002e2


Ive seen this a few times over the years and typically it means I need to repair the NTDS (and remove the log files) in the Windows\System32 directory


Tried to reboot into DSRM, Windows starts to load and then asks for my local admin password,  I put that in and it doesnt take it (Ive tried every variation of that password and nothing is taking).


So then I boot up from the Windows 2016 DVD Media, figuring I can just reset the password that way, no luck.  I can get to a command prompt X:\   but when I try to access my C:\ partition, there is nothing there (no C:\Windows or anyfolders) - except I can only see C:\ProgramFiles(x86).   Also, I cannot see my D:\ partition which has all of the clients data folders and files.


Even when I open notepad and launch explorer.exe there is really nothing there other than the C:\ProgamFiles(x86) - which is really from the D:\ drive because I always install 3rd party apps on a D: partition and never on the c:\


I did check that that hard drives are healthy and fine.  


I also tried 2 different 3rd party utility USB drives to boot from and change the local admin password but neither can find the OS


so now Im scratching my head.  Why is almost everything missing?  If the OS was missing why can I boot up into DSRM and when it lists my domain and when I switch to .\administrator for the user name it comes up with the local name of the server.   So Im thinking the OS is still there?


So has anyone experienced this and how can I get to "View" my C: and D: partition and all of their folders/files.  I find it very hard to believe that a power outage just wiped out everything.


Im wondering if I need to do a chkdsk and maybe repair any sectors?  again, the only error message I am receiving when booting up is what I mentioned in the beginning - 0xc00002e2


any thoughts or suggestions would be most welcome.


thank you so very much

Windows OSDellStorage HardwareWindows Server 2016Active Directory

Avatar of undefined
Last Comment
arnold

8/22/2022 - Mon
Shreedhar Ette

mkavinsky

ASKER
thank you for your response. as I mentioned in my statement I am unable to reset the DSRM password.  the C:\Windows directory is not there.   I also used 2 other third party utilities to try and reset and both cannot find the OS.  Please see my notes above

thank you
arnold

To Shreedar's point, the DSrM password by default is the admin account password  when the DC was promoted.
The change DSRM I think was more of a reminder to ..


I gather your issue is it is the sole DC in the environment.
Do you have more than one admin account?

Potentially, you need to boot it and adjust the registry to reinitialize ntds

Backups?
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
mkavinsky

ASKER
correct, its the admin account, there are no other local admin accounts.  Since AD isnt working I cannot use a domain admin account.  

typically for DSRM isnt there only 1 admin account anyway?  Ive never used more than one :)

there is another DC in the environment (its the backup DC).  yes I have a backup but really am trying to avoid going through that whole process.

Just need to know where did everything go? where is the C:\windows folder (and all the other folders), where is the entire D: partition with over 4TB of data?  that could not have possibly just wiped out (especially RAID1) and the drives are in a healthy state.
arnold

Dsrm password is set to the password of the local administrator account at the time of joining.
If you do not gave it, and you have a functional DC, the path is clear, and you'd be back in business.

You could try booting the system using install media, loading drivers, and then trying to use
Esentutil to check/repair c:\windows\ntds\ntds.dit

Note since you are not within a dsrm, ....
Jeff Glover

Actually, the DRsM password is set during DCPromo. It is not the default Administrator password on the machine before you promote it. That becomes the Administrator account in AD if it is the first DC in AD, if not, then it just goes away.
  If you have another DC in the environment, then you can just remove the Bad DC from AD and reinstall/Re-promote. There is no such thing in Server 2016 as a "Backup" DC. All DCs, except Read only DCs, are peers. As long as the good DC has DNS on it, you should be able to remove the bad one and reinstall it. I would make sure the Good DC passes DCDiag.
  When you reinstall, just format the C: drive partition. After install, you should be able to add the D: Partition back. As long as the Drives were not corrupted, the Data should be there.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
mkavinsky

ASKER
thank you for your suggestion.  And I have tried booting from the Windows server 2016 media disk but when I go to a command prompt and try to go to C:\windows - and as I have mentioned there is nothing there

so your suggestion of running the esentutil to check/repair wont work.  I  cannot "see" or access the C:\Windows folder.  its simply not there on C:\ProgramFiles(x86).   when I run a DIR on the C:\ partition, thats all I see.  when I switch to d:\ says its not there.  this is what I am not understanding is how is nearly everything gone? and after a simple power outage?  I still believe everything is there but "hidden", thats why I was asking if I should run a chkdsk or something?

Im not sure I understand your first statement:  
"If you do not gave it, and you have a functional DC, the path is clear, and you'd be back in business. "


thank you so much for your responses
mkavinsky

ASKER
Thank you Jeff

Correct, there are no more BDCs and PDCs, I was just stating I had a second server functioning only as a DC (with DNS).

My concern though is I dont even see the D:\ partition (where all the data is).  Im not opposed to just wiping that server and starting over (was hoping to avoid this) and just thinking there are some commands I can run to get into this server and repair the NTDS.   I dont believe the drives were corrupted but my only way I can check that was a health check on the hard drives(RAID1) and they are showing "healthy" (and both have green lights flickering in unison).  and the controller card and RAID configuration are still intact

I just want to make sure the data is still intact before I just blow away the C: partition.  

Again, was just thinking if I am missing something simple here.... if I am overlooking the obvious, lol.
ASKER CERTIFIED SOLUTION
Seth Simmons

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
mkavinsky

ASKER
Thank you Seth for your response as well.  Im now wondering if thats maybe the problem.  the storage driver is not loading thus not showing my volumes?  

Im attaching some screenshots here too so maybe this may help.

if I can see the login screen (during DSRM) then doesnt that mean it is finding the OS?
Ive also included the screenshot of what I see when booting from the media DVD and opening notepad.  it shows the C:\ drive (but I dont think thats the correct one, it shows the \ProgramFiles(x86) - which was on the D: partition.  Then if I can see that why do I need see the rest of that partition.  also, note the size of that C: partition - its 7.41GB  (not 7.5TB).
Then when I run a diskpart and list disk  - its only showing Disk 0 and 7.6GB?

Windows Server images.docx
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
SOLUTION
Scott Silva

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
mkavinsky

ASKER
Scott, I understand what you are saying about the file server not on a DC but please understand this is a 12 person environment that is a non profit and relys on Grants for any money they have, which leaves very little for IT.   The second DC was a repurposed server for them that I upgraded to 2019.  Again, this is a small environment with a very limited amount of resources

I appreciate your input, let me check out the article you are referring too.   I just am verifying the storage, its a PERC S130 SCSI 7.5TB so I can download the correct driver and add the driver via Windows boot media (or through Dell settings)
mkavinsky

ASKER
Seth,

I think youre onto something here.  I was able to load the Perc drivers (from USB drive I extracted them too) and I booted to the Windows media disk.  Had to go into "install" from the windows media disk so I can load the drivers.  Once I did I saw my system partition (although it was now F:\  and saw my data partition (which was D:\) and is now G:\ (thats not an issue)

so everything was there.  I went in and renamed the utilman.exe file and copied a new one over and then reset the administrator password.  Went back into DSRM mode and the password doesnt work (so I can correct the NTDS issue).  After rebooting it lost the driver I installed for the storage.  so the drives are gone again

Im doing something wrong because I dont want to have to reinstall this Perc S130 driver everytime I boot, its not keeping the driver.   I dont think I should have to go through the whole install of Windows Server 2016 again (that would overwrite everything and knock out AD and DNS on that server and maybe all the permissions to all the folders/files).

Im wondering if I can fix the NTDS (once I have the Perc drivers loaded) and booting to the Windows Media disc and try to correct it that way and not have to go into DSRM?   but there seems to be 2 issues here:

1) not loading the  Perc storage driver
2) to corrupted NTDS

thank you


SOLUTION
arnold

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
andyalder

>After rebooting it lost the driver I installed for the storage.

Way I read what you put you simply provided the driver to boot, you didn't install it.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
mkavinsky

ASKER
Thank you AndyAlder

so how do I install it then?  

this is what I did, I booted up from the Win2016 media disk, instead of clicking on repair I clicked "install"    
screen then prompted me and stated it could not find any storage devices, so i browsed out to the USB where I had extracted the Perc storage drivers.  Selected the correct driver and clicked "next"
Was then prompted for the OS to Install.  I cancelled that because I didnt want to install the OS.  From there I went back to "repair your computer", "troubleshoot" and then "command prompt".  Ran Diskpart and list disk and saw the 7.5TB storage,  then in my command prompt change to F:\ and there was the OS , switched to G:\ and there was all of the data files/folders

That probably just "temporarily" loaded the storage drivers.  not sure how I install them then.   Because there are 2 issues here, 1 - storage driver and 2 - NTDS

But its not going to to me much good fixing/repairing NTDS if I can even get the volumes loaded (if that makes sense) :)

So I need to install the perc drivers for good.  I think once I have that, I can move onto repairing the NTDS

hope that helps.

thank you again for everyones patience.  Second day on this now and its starting to mess with my head here :)


mkavinsky

ASKER
Thank you Arnold for your suggestion too.   But see my response to Andy.  I still think I need to get the storage drivers installed and loaded because im just going to have the same issue everytime I reboot.   even if I do repair the NTDS (which appears corrupted)

Im seriously contemplating what Jeff stated earlier (since I already have another DC) and that would be to wipe the OS on this downed server and reinstall Windows 2016. should leave the D:\ (data) intact. But I would have to reapply permissions to it again and re-created all the shares (there are quite a lot).

Clean up the metadata on the working DC and remove the one that gets rebuilt.  Then just re-join this server to the domain as a member server.

ugh... really want to avoid that (not looking forward to re-added all the permissions and sharing)  lol
SOLUTION
Scott Silva

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
mkavinsky

ASKER
Thank you Scott, I did use a purchased solution to reset the password and I got the message that no OS was found (was looking for the c:\windows directory).   Since I have a second DC I can still get into AD maybe I just reinstall the OS


​If the only way to keep the storage drivers installed and intact is to re install the windows 2016 server operating system, then I will just go that route.  Just want to make sure any verify with the experts here that may really be the best and only way to go?    All of you have been really helpful here and I am very appreciative of the time and suggestions presented. 



All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
andyalder

So your purchased solution doesn't have the drivers for the S130 fakeRAID controller, their support probably have a custom way to put it on their boot media. What product is it?
arnold

This makes little sense to buy a password resetter on a system that is a DC  (1 of 2) seems odd.
IT would be similar to having the entrance (1 of 2) to call out a lock smith as an emergency to gain access, while you have the keys to the other entrance.

Your issue is not the Drivers as you would have received a  different error, when the volume can not be accessed.
Your error is futher along, the windows OS is booting, but in the process of activating its role as a DC on the AD it sees the ntds.dit file as corrupt.

you can load the system through the life cycle F10 if not mistaken to deploy the OS. The life Cycle will present the requisite storage drivers to the OS installing software.The issue to consider is whether you have data on the server that you may need..... if backups are questionable.

The drivers can be loaded using the F6 option.
you just need to have the PERC drivers expanded and available on the USB, you can include the Network drivers just .....
mkavinsky

ASKER
Arnold, thank you for your response.  I had a purchased password reset tool that I have had to use in other instances so I did not purchase this specifically for this issue

Yes, I believe you’re correct that F10 is the lifecycle controller and can deploy the OS that way too 

I have verified that the data is still there (and I have a backup) 

I have the perc drivers on a usb drive and expanded.   So F6 during the boot up process will allow me to reinstall the storage drivers? (You mentioned network drivers too but I don’t think I have an issue with them) 
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
andyalder

I'd be wary about installing using the Lifecycle Controller, not sure whether it will preserve the data volume or not.
mkavinsky

ASKER
Good point.  I don’t want it to wipe and reconfigure the Raid 1 configuration.  Usually I just reinstall via the windows media disk.  I just want to make sure I’m taking the right path here before I pull the trigger 
SOLUTION
andyalder

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
arnold

Yes, boot up the system using the USB install media, when it tells you that it can not see media, go through the ADvanced option, and I think it there present an option to load drivers, which is when you would direct it to the USB portion where the drivers for the perc are, once it loads the drivers, it should give you an option to rescan the sotrage at which it should present you the RAID volumes your system has
at this point, you can select the location where the OS is installed, and get it to lnstall
leave the other volume alone, once the OS is setup. you can join into the domain as another DC and then look at configuring what it had as far as the other parittion if any...
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Scott Silva

Not every password reset disk can reset the Ad recovery password... Just a heads up...
It isn't really a regular account...
mkavinsky

ASKER
Arnold,
I am not seeing that option.  I am booting up to the windows 2016 media dvd. I select “repair your computer” then “troubleshoot” and then I’m in “advanced options”    There is no option to load driver

My options are:  system restore, system image recovery, startup repair (I tried that but nothing there), Command prompt and go back to previous build

Now if I selected “install” rather then “repair your computer” that’s where I get the pop up that no storage was found and from there I can access the USB drive where I have the driver but then as I’ve mentioned I cancel out of that OS install and I’m back at the same spot

so maybe I’m doing something incorrectly here where I’m not seeing the “load drivers” option?

thank you 
Scott Silva

As I posted previously you need to add the drivers to your rescue disk by making a custom copy.

https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/add-and-remove-drivers-to-an-offline-windows-image?view=windows-11


⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
mkavinsky

ASKER
Thank you Scott. But is it necessary that I need to create a custom rescue disk to load the perc drivers?   Would I get the same result if I reinstalled the OS and then when I’m asked for the drivers I point it to my USB where the drivers are extracted and then finish proceeding through the OS reinstall?   I’ve never used this procedure before so I’m just asking.   Thank you 
Scott Silva

I meant to do it with your purchased rescue disk to see if you could reset the ad recovery password...
Also you could use a rescue disk to back up the user files "just in case"...

SOLUTION
arnold

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
arnold

ust to be clear you did not lose access to the AD environment, you have one of two Dcs that crashed with corrupted NTDS DB on it.

The Computers, users and the remaining DC are functioning..... etc.

is the issue that the one that failed is the PDC, primary master?

If you decide to reinstall, you can use NTDSUTILS to seize the role on the existing functional DC.

install the OS on the failed DC, adn join it back into the AD as another DC.....
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
andyalder

Next time you buy a server get a real PERC rather than a Sxxx, (S stands for software although C for chipset would be more appropriate since it's the Intel chipset RAID with Dell's firmware on it). Then the drivers will already be on the Windows install media and included with most 3rd party tools since real PERCs are so popular. You could probably add a real PERC and use your current offline tool, the custom firmware on the chipset makes the disk metadata compatible with Dell's LSI based controllers.
mkavinsky

ASKER
Gentlemen,

Thank you all for all of your suggestions and potential fixes/solutions.  I was able to get everything back up and running.   So the solution was:
1) Perc storage drivers needed to be reinstalled
2) Reinstalled the Windows Server 2016 OS from the Media disc
3) all data was still there but just in case I was able to perform a robocopy to an external hard drive (wasnt necessary but always want to make sure nothing else went south).
4) After I got the server back up, performed a restore of the System State (AD, NTDS, DNS) but about halfway through the restore process  stopped, lost the OS and after other attempts needed to reload the OS again.
5) Did a manual cleanup of the metadata, seized the FSMO roles to the other DC.
6) joined the rebuilt server as a member server/file server (taking Arnolds advice and not letting it be a domain controller anymore).
7)re-sharing network folders/apply permissions and final clean up

But all in all got it all back up and running.

Oh, in case anynone was wondering, I did open a support case prior to reaching out to the experts.  Took MS support 4 days! to get back to me.  I was promised a 2 hour response time, I get they too have been overwhelmed but 4 days to even get an email from paid support is not good.  So thank you again everything that helped contribute and the promptness of your replies.  Team effort!!  thank you again
arnold

When you have multiple DC's, you should not be restoring DC, there is no need.
You can as you have done:seize the roles, readd the newly reinstalled system as another DC to reestablish redundancy.
The two will sync..

You restore of DC creates a potential conflict.

You do not want to run with a single DC, consider what state will you be in if the DC you had issues with was the only one in the environment?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.