Mango-Man
asked on
Server 2012 Hyper-V 3 backup restore failed, error compacting VHD, VM missing
Hi All,
This morning we had a small incident grow into a bigger one, and 5 hours later my Microsoft Support case still isn't open due to location issues, so I am on here seeking some help ASAP.
We recently migrated a client to a new Server 2012 backup solution, single physical host, 2 VMs, (VM1 = ADS, DNS, File server, as well VM2 = SQL LineOfBusiness server), and there is still a legacy SBS2003 physical box on the network that still currently holds the FSMO roles and is acting as a DC and DNS server as well (will be physical backup DC eventually).
All has only been in a few weeks, and the Server 2012 native backup solution has been used, installed on the host, and set to back up itself and the 2 VMs for, for file, volume and bare metal, to an external 2.0 TB USB 3.0 drive.
Today, a user deleted a large project directory by accident. I thought no problem, and went into the Backup on HOST. Backup had previously been showing no errors a week or so back, but was now showing as "complete with errors – there was a failure compacting the VHD" or something to that effect. (This may be since I had to increase one of the VM's VHD sizes last week).
Browsing the backup, trying to use file level recovery, the latest backup was not showing either of the VMs in the available backup tree. Checking the error more carefully, the backups were listing as completed but with no logical path listed for the VMs (V: drive on host holding the VMs and VHDs).
**************************
Backup of volume \\?\Volume{29d7b669-28fd-1
Backup of volume C: succeeded.
Backup of volume V: succeeded.
Application backup
Writer Id: {66841CD4-6DED-4F4B-8F17-F
Component: 7FB96AAF-729C-49AB-AF91-84
Caption : Backup Using Child Partition Snapshot\VM1***S
Logical Path:
Writer Id: {66841CD4-6DED-4F4B-8F17-F
Component: DA0EAE31-0277-4CFC-ADD7-6B
Caption : Backup Using Child Partition Snapshot\VM2****W
Logical Path:
Writer Id: {66841CD4-6DED-4F4B-8F17-F
Component: Host Component
Caption : Host Component
Logical Path:
*-------------------------
**************************
So basically i couldn't perform a simple file/folder recovery. For the life of me, I couldn’t browse or see the deleted files... strike 1.
(It then occurred to me to try shadow copies (not big I/O volume here), which I thought were enabled, to find out they were not.. strike 2)
Normally we’d have a remote backup in place, and it was on my immediate snaglist of things being finished off, but you know murphy’s law, and with it being a brand new deployment… strike 3.
So, here’s where I made my major gaff – with the customer wanting things back urgently, and it being first thing in the morning, with no major changes, I thought quickest resolution would be zapping the whole VM back, which was still showing as available, and “finished/complete”, despite the compaction error. I should know a lot better by now, but didn’t grab a file copy of the server data first.
Running the restore job on the whole VM1 server now failed a minute after starting, stating “Detailed error – the parameter is incorrect”. Useful.
Only now the VM has disappeared and won’t start…. The VHD is still visible, but the VM config files have vanished. So the whole of VM1 (DC, ADS, File server) is now down, rather than just a directory.
I’ve read how I should be able to delete all Virtual Networking config from the server, and create a new VM to import the old VHD to, but wasn’t sure of the ramifications on a DC.
As the HOST backup is still OK, am I ok to restore the VM config files from midnight backup last night, and try and start the machine using them, or are they time stamped to the VHD, so that the current, 6 hour older VHD, will not work?
I’m unsure of my ground here on Hyper-V, and MS STILL haven’t got back to me, so all help/advice appreciated at this stage. (I’ve made a backup copy of the VHD to an external drive already).
This is an URGENT/Network down type situation...
Many thanks
Dom
This morning we had a small incident grow into a bigger one, and 5 hours later my Microsoft Support case still isn't open due to location issues, so I am on here seeking some help ASAP.
We recently migrated a client to a new Server 2012 backup solution, single physical host, 2 VMs, (VM1 = ADS, DNS, File server, as well VM2 = SQL LineOfBusiness server), and there is still a legacy SBS2003 physical box on the network that still currently holds the FSMO roles and is acting as a DC and DNS server as well (will be physical backup DC eventually).
All has only been in a few weeks, and the Server 2012 native backup solution has been used, installed on the host, and set to back up itself and the 2 VMs for, for file, volume and bare metal, to an external 2.0 TB USB 3.0 drive.
Today, a user deleted a large project directory by accident. I thought no problem, and went into the Backup on HOST. Backup had previously been showing no errors a week or so back, but was now showing as "complete with errors – there was a failure compacting the VHD" or something to that effect. (This may be since I had to increase one of the VM's VHD sizes last week).
Browsing the backup, trying to use file level recovery, the latest backup was not showing either of the VMs in the available backup tree. Checking the error more carefully, the backups were listing as completed but with no logical path listed for the VMs (V: drive on host holding the VMs and VHDs).
**************************
Backup of volume \\?\Volume{29d7b669-28fd-1
Backup of volume C: succeeded.
Backup of volume V: succeeded.
Application backup
Writer Id: {66841CD4-6DED-4F4B-8F17-F
Component: 7FB96AAF-729C-49AB-AF91-84
Caption : Backup Using Child Partition Snapshot\VM1***S
Logical Path:
Writer Id: {66841CD4-6DED-4F4B-8F17-F
Component: DA0EAE31-0277-4CFC-ADD7-6B
Caption : Backup Using Child Partition Snapshot\VM2****W
Logical Path:
Writer Id: {66841CD4-6DED-4F4B-8F17-F
Component: Host Component
Caption : Host Component
Logical Path:
*-------------------------
**************************
So basically i couldn't perform a simple file/folder recovery. For the life of me, I couldn’t browse or see the deleted files... strike 1.
(It then occurred to me to try shadow copies (not big I/O volume here), which I thought were enabled, to find out they were not.. strike 2)
Normally we’d have a remote backup in place, and it was on my immediate snaglist of things being finished off, but you know murphy’s law, and with it being a brand new deployment… strike 3.
So, here’s where I made my major gaff – with the customer wanting things back urgently, and it being first thing in the morning, with no major changes, I thought quickest resolution would be zapping the whole VM back, which was still showing as available, and “finished/complete”, despite the compaction error. I should know a lot better by now, but didn’t grab a file copy of the server data first.
Running the restore job on the whole VM1 server now failed a minute after starting, stating “Detailed error – the parameter is incorrect”. Useful.
Only now the VM has disappeared and won’t start…. The VHD is still visible, but the VM config files have vanished. So the whole of VM1 (DC, ADS, File server) is now down, rather than just a directory.
I’ve read how I should be able to delete all Virtual Networking config from the server, and create a new VM to import the old VHD to, but wasn’t sure of the ramifications on a DC.
As the HOST backup is still OK, am I ok to restore the VM config files from midnight backup last night, and try and start the machine using them, or are they time stamped to the VHD, so that the current, 6 hour older VHD, will not work?
I’m unsure of my ground here on Hyper-V, and MS STILL haven’t got back to me, so all help/advice appreciated at this stage. (I’ve made a backup copy of the VHD to an external drive already).
This is an URGENT/Network down type situation...
Many thanks
Dom
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
I would create new Virtual Machine configuration, and add the existing good VHD that you have to this configuration, and start the VM.
ASKER
ok, thanks, that worked fine, with a little fiddling on the network side recreating virtual NICs and attaching to virtual switch. I suspected it would, but wanted to be certain with it being a DC, as haven't had to consider it in a production environment before.
So i'm back to my missing directory...
now does anyone have any idea on how I might be able to dig deeper into this backup failure issue? Is there maybe a 3rd party tool that also examine the backup VHD or files and see what it can recover?
i've already used a couple of undelete utilites to pull back what I could, but some stuff already overwritten - looks like we may be needing the backup in some way...
So i'm back to my missing directory...
now does anyone have any idea on how I might be able to dig deeper into this backup failure issue? Is there maybe a 3rd party tool that also examine the backup VHD or files and see what it can recover?
i've already used a couple of undelete utilites to pull back what I could, but some stuff already overwritten - looks like we may be needing the backup in some way...
there are many utilities, that can mount VHDs, do you have a good VHD backup?
ASKER
this is the problem:
The Native server 2012 backup was being used from the host, to a dedicated bacup drive - it was listing as Complete, but with errors compacting the VHD backup file (which I think is only used for backing up the VMs). SO now have a dedicated Windows backup drive, holding what is the native 2012 backup, which i need to explore or try and mount. Whilst I know there are dozens of tools to to mount VHDs, does anyone have experience or knowledge of something that can explore the Native 2012 backup media, as I simply don't have enough time to trial and error a dozen different apps here, so I'm offering it up for the remaining points (split) on the rest of this question! :)
(to be fair, I should have made clear the points split in the original post - 50% on whether to go ahead and create new VM, as answered, and 50% for helping me explore/get into the damaged backup)
Thanks
The Native server 2012 backup was being used from the host, to a dedicated bacup drive - it was listing as Complete, but with errors compacting the VHD backup file (which I think is only used for backing up the VMs). SO now have a dedicated Windows backup drive, holding what is the native 2012 backup, which i need to explore or try and mount. Whilst I know there are dozens of tools to to mount VHDs, does anyone have experience or knowledge of something that can explore the Native 2012 backup media, as I simply don't have enough time to trial and error a dozen different apps here, so I'm offering it up for the remaining points (split) on the rest of this question! :)
(to be fair, I should have made clear the points split in the original post - 50% on whether to go ahead and create new VM, as answered, and 50% for helping me explore/get into the damaged backup)
Thanks
ASKER
I've found a solid backup from Jan the 3rd, so not all lost anyway. Just need to mount the VHD tomorrow.
Investigating why the backups are failing on the VHD compaction, it appears the 2 server VM VHDs are both listed as dynamic, despite definitely being specified as fixed disks during setup (backups started failing after I expanded on of them it seems). I'm gong to convert them tomorrow, after taking manual copies, which seems sensible to me given lack of working local backup currently. I've read to set the VHDs to read only before copying, and then setting back afterwards before converting.
Just want some confirmation that this would be a good plan...?
Investigating why the backups are failing on the VHD compaction, it appears the 2 server VM VHDs are both listed as dynamic, despite definitely being specified as fixed disks during setup (backups started failing after I expanded on of them it seems). I'm gong to convert them tomorrow, after taking manual copies, which seems sensible to me given lack of working local backup currently. I've read to set the VHDs to read only before copying, and then setting back afterwards before converting.
Just want some confirmation that this would be a good plan...?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Forgot about this question apologies.
Converting the VHDs back to fixed disk did in fact rectify the backup issue.
hanccoocka, thanks for the help!
Converting the VHDs back to fixed disk did in fact rectify the backup issue.
hanccoocka, thanks for the help!