Windows 2012 R2 - inaccessible boot device

Steve Knight
Steve Knight used Ask the Experts™
on
Got a bit of a problem with a Windows 2012 R2 box and would appreciate any suggestions.  it isn't a massively vital system (internal instant messaging server) and I sill have a few other potential restore options before rebuilding but I wonder whether I have missed something blindingly obvious and would like to see if can solve issue, apart from anything else before rebooting the other 8 or so VM!

This is running on VMware ESXi with two hosts running to a iSCSI SAN.  There is also a DR site with VM replication setup

This started at the weekend with a routine reboot.  The server hadn't been rebooted for a few months, possibly 4-5 months and it had a small number of updates + a load of optional ones.

I didn't create a VM snapshot before it rebooted, apart from anything else it had just backed up and replicated to DR site and was routine updates.

But... it disappeared, checking the VM console it was in a cycle of failed boot and automatic repair.  It shows "inaccessible boot device" but just that, no code like I would normally expect to give a hint.

At that point I took a snapshot which I reverted to at various points before trying different repairs and restore.

Safe mode, command prompt, last known good etc. all failed the same of course
The command prompt repair had an issue in that the local admin password for this box did not match documentation and would not work.
So after a boot off Windows 2012 R2 DVD.  Login to command prompt that way.
        Carried out checks with command line tools, Bcdedit, bootrec etc.  All looks OK, partitions look OK etc. right drive active, BCD looks OK etc.
        Looked with DISM etc. to see what updates were put on. = approx. 4 .NET framework updates.  Multiple other updates pending but not yet installed

kb2894856 = net framework 4.5
KB3010788 = OLE on Windows 7?
KB3032663 = net framework 4.5.2
KB3037579 = net framework 4.5.2
kb3045717 = Narrator
KB3094486 = KDS

        Pending updates - tried clearing using online suggestions removing pending XML files, and registry entries
        Reset local admin password using Linux NTPASSWD boot disc.  It advised it thought password might already be blank (I later tried blank password which did not work either)
        So then command prompt repairs same as above using built in recovery command prompt

        Boot logging did not help, did not seem to drop the log file.

        No obvious memory dmp files.

Then tried a restored using Symantec SSR boot CD restoring C: and System State from backup few days before any updates (19/9).

Still the same.

I cloned VM to new one and moved to other host
        Not surprisingly the same...
        Restored latest backup from SSR using boot CD - system state and C: drive.  Same
        Restored oldest backup from SSR using boot CD. Same

Today: Built fresh new empty VM
        Restored in full newest backup before reboot - same


So... any suggestions I might be missing?

I have two other options open to me I think:

1. Use recovery at the DR site and see if that boots and bring back to main site.
2. Build new OS, installs apps and data....

I've not had to recover from a VMware replication before as previously used Storage based replication and for small companies without a SAN Hyper-V replication methods.  It looks like once I do a recovery the option of going back to recover again will be gone.  I am not convinced the recovered machine will be any different though will have a few snapshots to go back through trying.

thanks

Steve
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Philip ElderTechnical Architect - HA/Compute/Storage

Commented:
If the partitions and data are all there when the VM gets booted via .ISO file, then one place to look is in the settings for the VM in VMware. Has anything changed there? Is the boot order still correct and Drive 0 still the OS VMDK file?
Philip ElderTechnical Architect - HA/Compute/Storage

Commented:
Also, you could try to rebuild the boot configuration database in-guest. So, boot a Windows installer .ISO file and at CMD:
bootrec /rebuildbcd

Open in new window

VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
Restore from Backup, there have been a few on here in the past few weeks.....you'll waste time try to find a cure, the disk is corrupted, or likely registry, unless you've changed the hardware which is highly unlikely!

Your restore from backup does not work, when was the last time this was tested ?
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
Thank you both.  I have tried the bootrec command already at some point though I can soon try it again.

I think I am write in thinking MBR must be right because we've got past that, Boot record should be OK as we have hit the Windows logo and spinner and must be at the Kernel stage which made me think logically would be storage drivers etc.

I've had ones before with storage controller issues but would have seen a code, we just get this:

Win logoyeah thanks Microsoft

Yeah backups... they are carried out using Symantec system recovery 2013 from within the VM to images.  I've got daily images going back 2 weeks and restoring those over the VM or into a fresh VM with same virtual hardware and get the same boot issue.

I wouldn't normally be too worried about an isolated machine but have 8 other 2012 R2 servers built by the same people that I inherited on the same VM infrastructure, same AV etc. which I am not keen on rebooting just at the mo!

My next option then might be the replications to DR site using VM replication appliance but not a procedure I have done before myself, looks easy enough but I've not seen a document yet identifying best practice - i.e. whether I can preserve the state of the replication, copy it and recover that or whether the only option is to do a "recover" and then hope that boots and I presume clone while on the snapshot that want to keep into a fresh VM.

steve

Author

Commented:
Sorry crossed over there I think. As to the backups.... the backups have not reported any errors and I have tested in terms of restoring files from every VM and proved that works fine (and still do) and the disks and partitions all show as restored OK to a new VM too which behaves the same as the original so I suspect backup is working but that the OS had an issue that was going to crop up after next reboot and nothing to do with (those) updates and is probably on all the backups too.

Would appreciate any advice you could give please on bringing back using the DR replica, though gut feeling is going to end up with the same issue it may well be a case of build a new OS, apps and restore the data back (which also appears OK).

thanks
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
You are not taking advantage of the VMs, by backing them up as a VM, rather than just files and folders! (just like a physical PC) using Symantec System Recovery, not sure why you are not backup up Virtual Machines as they should be.

Did you break the DR replication otherwise replicas will also be likely the same.

Is the VMware vSphere Replication ?

Author

Commented:
I know.  That is what I have inherited, on the (lengthy) list to change within the time I have available to work on these systems.  Yes it is VMware vSphere replication, with a replication appliance machine running.

I paused the replication as soon as I knew there was an issue so we have 7 earlier replications:

Outgoing replication setting - paused.  Server was rebooted after the evening of 21/9
Not an area I have used like I say for recovery but from what i read if I set off recovery I'll end up with a VM with multiple snapshots.  I assume then best would be to clone the machine to try a boot and with NIC offline.

Steve
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Okay so I assume you don’t know how to restore the earliest replica

It’s good replication has been paused because based on config the earliest replicas can just get erased

Author

Commented:
Yes with only few days worth I thought would be best.  Will look at Recovery tomorrow I think.

Steve

Author

Commented:
I've got time there on Tuesday and from what I can see do it as:

1. recover the most recent entry from DR site.  That should leave me with a VM with previous replications as snapshots
2. copy the VM
3. Boot that up on DR site.  If it does boot, do test reboots, add the missing updates and boot again etc. and try and break it!
4. if not revert to earlier snapshot and try the same again.

If/when it works copy the VM back to main site, clean up and set replication back up.

Does that sound sensible, I suppose step 2 isn't actually necessary but feels better working on a copy.

Steve
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Yes, that's correct, but I would restore the oldest replica.

Author

Commented:
Thank you. Given up now, the recovery process worked perfectly but all 6 replicas held the same issues as did 2 weeks worth of backups. I restored other VM from scratch using same procedure and it was fine so clearly that OS broke itself weeks or months ago.

Time for new OS, app installs and recover data from the backups...

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial