Established 2019 Hyper-V VMs suddenly won't boot - need to fix

Fred Marshall
Fred Marshall used Ask the Experts™
on
I set up 3 Windows Server 2019 Standard systems.  
On each machine, the first OS install was set up as a Hyper-V manager.  
The second (VM) OS install was a Server VM for a DC.  
I plan on a 3rd VM OS install for a Server VM perhaps for file serving - yet TBD.

I've been in and out of these machines fairly regularly but have been away from them for the last month.
Now, it appears, none of the VMs will boot - not on any of the 3 systems.
The VMs had been running but I believe I encountered them today Stopped.
(The Hyper-V systems were running).
And, oh yes, there was a power outage a few days ago.  But I did try to take down the systems gracefully.
So that's probably why the VMs weren't running.
And, maybe the power cycle triggered an update.

Anyway, The message says that a boot device wasn't found and seems to go to PXE.
Then it suggests that the VM be configured.

I have a bit of investment in the configurations but haven't yet reached the point where I've set up the backups.
I'd like to keep them as-is.

What is the recommended procedure to get these VMs booting?
From what I've read, it seems to be possible but most things I've seen seem rather tentative or on a slightly different issue.

On a slightly different subject, why the heck did I decide on VMs IF the 3rd install is still tentative?  
This doesn't seem robust enough for real time.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Philip ElderTechnical Architect - HA/Compute/Storage

Commented:
Right click on the VM in Hyper-V Manager and EDIT.

Flip Secure Boot off (untick) then Apply and OK.

Try and boot.

Author

Commented:
Philip Elder:  Thanks!
I did that but still have issues.
I've tried them both with Secure Boot (which is where they started today) and NOT Secure Boot.

I noticed that the boot sequence didn't necessarily start on the hard drive .vhdx file so I made all of them with this the first on the list.
I hoped that this would help with some of the boot issues.

I notice that using Secure Boot, this happens:
System #1 Description: EFI SCSI Device  Value: C:\Users\Administrator\Downloads\DC1-RAY Virtual Machine.vhdx
1. SCSI Disk (0,0) the boot loader did not load an OS
2. SCSI DVD (same)
3. Network adapter.  A boot image was not found.  ... well, there is none there.

System #2 EFI SCSI Drive: Value: C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\DC2-LB Virtual Machine.vhdx
Eventually get message:
Microsoft Hyper-V UEFI
1. SCSI Disk (0,0) the boot loader did not load an OS
2. Network adapter.  A boot image was not found.  ... well, there is none there.

System #3 EDescription: EFI SCSI Device Value: C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\DC3-SB Virtual Machine.vhdx
Eventually get message:
Microsoft Hyper-V UEFI
1. SCSI Disk (0,0) the boot loader did not load an OS
2. Network adapter.  A boot image was not found.  ... well, there is none there.

Then, without Secure Boot:

System #1 Description: EFI SCSI Device  Value: C:\Users\Administrator\Downloads\DC1-RAY Virtual Machine.vhdx\
Initially comes up with black screen:
"Press any key to boot drom CD or DVD"
and then:
1. SCSI Disk (0,0) the boot loader did not load an OS
2. SCSI DVD (same)
3. Network adapter.  A boot image was not found.  ... well, there is none there.

System #2 EFI SCSI Drive: Value: C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\DC2-LB Virtual Machine.vhdx
Eventually get message:
Microsoft Hyper-V UEFI
1. SCSI Disk (0,0) the boot loader did not load an OS
2. Network adapter.  A boot image was not found.  ... well, there is none there.

System #3 EDescription: EFI SCSI Device Value: C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\DC3-SB Virtual Machine.vhdx
Eventually get message:
Microsoft Hyper-V UEFI
1. SCSI Disk (0,0) the boot loader did not load an OS
2. Network adapter.  A boot image was not found.  ... well, there is none there.

Each one says "No operating system was loaded. etc."
Top Expert 2016
Commented:
That would imply to me that the .vhd(x)'s are corrupted. Either the file or the contents of the vhd(x)
run chkdsk /f on the drives that the virtual machines hard drives are located.
try and mount the vhd(x)'s into the windows file system on the host o/s
it they mount then put them online and see if you get any errors i.e. RAW partitions
check in diskmgmt.msc make online/offline assign drive letters etc
https://www.screencast.com/t/SenDsU6QWzm
Become a CompTIA Certified Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

Author

Commented:
I've not done this before so I'm really a noob.
I see TWO VHDX files in
C:\Users\Public\Documents\Hyper-V\Virtual hard disks\
One is called:
servername Virtual Machine.vhdx and has a size of 4,096KB
The other is called, I believe:
servername.VHDX and has a size of around 130 GB.
... something like that.

Now, I ran chkdsk c: /f on each one and on two of them the larger of the two files is gone.
Somehow I think that was the more valuable of the two files in some sense.
So, it's more than a bit troubling.
I perhaps should have done one at a time....

Now that I think about it, I was shutting the machines down one at a time as fast as I could in preparation for the power outage and only surely got the one done in time.  That's the one with the larger file remaining.
It's concerning that a power failure would have caused this although I don't deny strange things like this do happen.  

The good news may be that these 3 systems were virtually identical with the exception of IP addresses.  So, I may be able to get the things back from the one that remains.  It still won't boot though.  

I have tried mounting the file and it says it's in use.
Top Expert 2016

Commented:
you can't mount the file if hyper-V virtual machine is running that uses the file. Since 1 file is now gone. if you go into that vm's settings is the missing file referenced there?

Author

Commented:
David:  I'm not sure what you mean as I've not dealt with these things much at all.  
But, if I understand your meaning, the settings don't show the filename and browsing from there can't get to it as it's missing.

I believe the hope is to resurrect the one file that remains in existence.  But how?
Philip ElderTechnical Architect - HA/Compute/Storage

Commented:
In Generation 2 VMs there is actually a .EFI file that is first in the boot order as I recall.

If there is a need to get things going then create new VM shells and attach the existing VM VHDX file(s) and boot.

Clean-up networking and re-seat the correct IP address then reboot the VM to get things going.

Author

Commented:
This is still a bit beyond me.
These are Gen 2.
I found a bunch of .efi files but all are dated 9/15/2018 and nothing much interesting re: the VMs I had created.

NB: there is but one system that still has a large VHDX file.
So, presumably that file presents an opportunity to fix at least that one.
It won't Mount.
That would be very useful as the other machines are virtually identical except for IP addresses and some site details.
And, they would be on other platforms entirely.

How to fix that one?

..create new VM shells?

I also see small VHDX files of size 4,096.
Top Expert 2016

Commented:
the 4K files are probably just empty vhd's You have to go into the virtual machines settings. and identify the vhd's that are required for each vm.
Perhaps booting each vm using installation media or windows PE iso will allow you a chance to access the virtual drive and run the checks you would do with a physical machine.

Having proper backup's would have eliminated your current situation. We won't cry over spilt milk.

When you say there wasn't enough time to shutdown some vm's is interesting. You could have "saved" the vm's.. it is faster than a shutdown.

Having corrupted files begs the question on your file system as a whole. I've had quite a few power outages lately and the VM's were not damaged at all.  Perhaps because they are in a battery backed up RAID has prevented this problem.
Perhaps you have turned off windows write-cache buffer flushing?2019-08-13_21-52-49.png
Either way your entire filesystem is at risk given your current policies.  If the vhd(x)'s were damaged begs the question of what other files were damaged.

Author

Commented:
David Johnson:  Yes, I agree with all of that.  I'm thinking of redoing the entire installation on at least two of the systems that didn't get shut down in time.  The one, I believe, did get shut down in time and yet, the VHDX won't boot.
What I believe happened with the other two is that they had been ordered to Stop and the power outage occurred while this was still in process.

I surely agree with the backups comment.  It's like it should be nearly "step 1" in view of what I encountered here.

Just to be clear, which write cache setting are you suggesting be used?

When I Start the one\, I first get a black screen that says:
>>Start PXI over IPv4             .... which I don't get really.
A nicely fonted Hyper-V(tm)
Both in a black window.
Then it changes to a white window saying:
1. SCSI Disk (0,))
   The boot loader did not load an operating system.
2. Network Adapter (########)
   A boot image was not found.  .... which is no surprise as there isn't one AFAIK.

The question remains, can the existing large VHDX be repaired?

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial