academynetworks
asked on
Hyper-V guest 'crashes' every time it is backed up.
Hi,
We have a fairly new Hyper-V installation and the guest server 'crashes' every time it is backed up.
The backup is a simple USB backup disk - they have a couple and they swap them out and it's set to run a full host, guest and bare metal backup every night.
The VM is a SQL guest and the management company for the SQL app requested that both the system and data drives were setup as fixed disks so it was setup with around 133GB on the system and 681GB on the data (or at least this is how they're shown in Explorer - Hyper-V sees them as 127GB and 650GB). The partition that houses them is 795GB in total and shows a little over 17GB free with the above usage. Note that this server is built with SSD's in RAID 10 hence the lack of disk space.
I have tested and confirmed that it is the backup and specifically the shadow copy part of the backup that causes the issue. When it is disabled or even when the backup integration option is unticked, it doesnt crash. When the backup runs we get some errors on the host event logs:
My question is obviously 'how do I fix this' as every morning the guest is powered off, but also, when using fixed VHDX's is there guidelines for how much data you need to leave free for shadow copy to run? I'm sure we have other systems out there with full data drives that dont crash the guest every time the backup runs. And its worth noting that shadow copy is actually disabled on the D drive so presumably this is literally shadow copy being required to run the backup.
Obviously given the 17GB free on the D drive, I dont have a lot of space to re-size the disks if its a job that will essentially rebuild the VHDX. I could do it via a USB disk but ideally I dont want to shrink the drive.
I am seriously contemplating creating some kind of script to shut down the VM for the backup then power it on after. Do you think this is worth pursuing? I might try it manually one night to see if it works. Obviously i'd prefer to actually fix the issue rather than jury rig it!
Many thanks
We have a fairly new Hyper-V installation and the guest server 'crashes' every time it is backed up.
The backup is a simple USB backup disk - they have a couple and they swap them out and it's set to run a full host, guest and bare metal backup every night.
The VM is a SQL guest and the management company for the SQL app requested that both the system and data drives were setup as fixed disks so it was setup with around 133GB on the system and 681GB on the data (or at least this is how they're shown in Explorer - Hyper-V sees them as 127GB and 650GB). The partition that houses them is 795GB in total and shows a little over 17GB free with the above usage. Note that this server is built with SSD's in RAID 10 hence the lack of disk space.
I have tested and confirmed that it is the backup and specifically the shadow copy part of the backup that causes the issue. When it is disabled or even when the backup integration option is unticked, it doesnt crash. When the backup runs we get some errors on the host event logs:
There was insufficient disk space on volume D: to grow the shadow copy storage for shadow copies of D:. As a result of this failure all shadow copies of volume D: are at risk of being deleted.
The shadow copies of volume D: were aborted because the shadow copy storage failed to grow.My question is obviously 'how do I fix this' as every morning the guest is powered off, but also, when using fixed VHDX's is there guidelines for how much data you need to leave free for shadow copy to run? I'm sure we have other systems out there with full data drives that dont crash the guest every time the backup runs. And its worth noting that shadow copy is actually disabled on the D drive so presumably this is literally shadow copy being required to run the backup.
Obviously given the 17GB free on the D drive, I dont have a lot of space to re-size the disks if its a job that will essentially rebuild the VHDX. I could do it via a USB disk but ideally I dont want to shrink the drive.
I am seriously contemplating creating some kind of script to shut down the VM for the backup then power it on after. Do you think this is worth pursuing? I might try it manually one night to see if it works. Obviously i'd prefer to actually fix the issue rather than jury rig it!
Many thanks
ASKER
Hi David,
Thanks for the response but I don't *think* that is it. To be clear, it's not a USB stick, it's a 2TB External USB drive. We basically use them as tapes as they're about the same size, around the same cost and't they don't require an expensive tape drive that can easily fail. We use them with the built in Windows Backup as it costs nothing, does bare metal backup and recovery and it generally works very well.
I just gave the backup disk a drive letter (as Windows Backup removes the drive letter) to view it and whilst it does only have 400GB free, the backup should cycle old backups as it needs. We use this system at a lot of sites and it works (generally!).
Also the errors state that it's drive D (drive D being the VM storage drive for the hypervisor) shadow copy that cannot grow so I dont think it's related to the backup drive at all. I think Windows backup is trying to use drive D as shadow copy storage for the backup but if thats the case, how the hell to you plan the layout of a drive if there are no guidelines as to how much the backup needs to run? I.E. does it need a quarter of the total disk size, a half, twice as much? And it wouldnt seem to make sense for you to always need to keep half the disk free just for the shadow copy backup to work.
Still scratching my head over here...!
Thanks
Thanks for the response but I don't *think* that is it. To be clear, it's not a USB stick, it's a 2TB External USB drive. We basically use them as tapes as they're about the same size, around the same cost and't they don't require an expensive tape drive that can easily fail. We use them with the built in Windows Backup as it costs nothing, does bare metal backup and recovery and it generally works very well.
I just gave the backup disk a drive letter (as Windows Backup removes the drive letter) to view it and whilst it does only have 400GB free, the backup should cycle old backups as it needs. We use this system at a lot of sites and it works (generally!).
Also the errors state that it's drive D (drive D being the VM storage drive for the hypervisor) shadow copy that cannot grow so I dont think it's related to the backup drive at all. I think Windows backup is trying to use drive D as shadow copy storage for the backup but if thats the case, how the hell to you plan the layout of a drive if there are no guidelines as to how much the backup needs to run? I.E. does it need a quarter of the total disk size, a half, twice as much? And it wouldnt seem to make sense for you to always need to keep half the disk free just for the shadow copy backup to work.
Still scratching my head over here...!
Thanks
The error is stating that there is not enough free space to grow the snapshot. That's the problem.
We use fixed VHDX files for the guest OS VHDX and any data VHDX that is 250GB or smaller. Anything larger than that gets created last, that is after all of the fixed VHDX files, and is set to Dynamic. There is no real reason to run with fixed VHDX files across the board when set up this way as fragmentation over time is not an issue.
Fragmentation in a SSD setting is not really a problem either.
If doing host level backups as seems to be the case here Veeam is way better than the native Windows Server Backup in most ways.
I have two very thorough EE articles on all things Hyper-V:
Some Hyper-V Hardware and Software Best Practices
Practical Hyper-V Performance Expectations
We use fixed VHDX files for the guest OS VHDX and any data VHDX that is 250GB or smaller. Anything larger than that gets created last, that is after all of the fixed VHDX files, and is set to Dynamic. There is no real reason to run with fixed VHDX files across the board when set up this way as fragmentation over time is not an issue.
Fragmentation in a SSD setting is not really a problem either.
If doing host level backups as seems to be the case here Veeam is way better than the native Windows Server Backup in most ways.
I have two very thorough EE articles on all things Hyper-V:
Some Hyper-V Hardware and Software Best Practices
Practical Hyper-V Performance Expectations
ASKER
Thanks for your comments Philip.
I don't suppose you have any idea of a rule of thumb when it comes to MS backups and required free disk space? I'm struggling with the idea that Microsoft doesn't at least document something about exactly how VSS uses available disk space. I.E. If VSS requires lets say half the size of a VHDX to back it up, then surely this must be documented. Otherwise it's just guesswork and obviously we don't like guesswork in IT! I looked through your links which are quite helpful but obviously nothing related to this specific issue.
The problem that we have now is that we've sold a solution and unfortunately, in this instance, it isn't working well so it's going to be difficult for us to now recommend more disk space or an expensive backup solution as we're likely going to have to pay for it.
My thinking is that if MS supply a backup solution that requires X amount of disk space to even function, then surely that fact should be documented somewhere? Once we know that we can start trying to sort this mess out...
I might just open an MS support ticket but thought i'd try here first.
Cheers
I don't suppose you have any idea of a rule of thumb when it comes to MS backups and required free disk space? I'm struggling with the idea that Microsoft doesn't at least document something about exactly how VSS uses available disk space. I.E. If VSS requires lets say half the size of a VHDX to back it up, then surely this must be documented. Otherwise it's just guesswork and obviously we don't like guesswork in IT! I looked through your links which are quite helpful but obviously nothing related to this specific issue.
The problem that we have now is that we've sold a solution and unfortunately, in this instance, it isn't working well so it's going to be difficult for us to now recommend more disk space or an expensive backup solution as we're likely going to have to pay for it.
My thinking is that if MS supply a backup solution that requires X amount of disk space to even function, then surely that fact should be documented somewhere? Once we know that we can start trying to sort this mess out...
I might just open an MS support ticket but thought i'd try here first.
Cheers
Use the following to clean-up VSS snapshots:
That should clean-up the error.
What backup product?
vssadmin delete shadows /allThat should clean-up the error.
What backup product?
ASKER
Thanks, pretty sure I've already tried that but will give it another go. Not sure if I noted that shadow copies are actually disabled on that volume?
Interestingly, the below screenshot is of a system setup in almost exactly the same way - fixed disks and with less free space on the data drive and yet this one backs up perfectly. Different version of server though - this is 2012 R2 and the problem machine is 2016. This makes me feel it isn't normal behavior for it to run the disk out of space then crash the VM if there isn't 'enough' disk space on the drive in question.
This is Windows backup - the built in backup that comes with the OS. We've used it for years just for the onsite backup and never usually get any issues with it (we also usually use a cloud based backup too although this particular customer declined that option).
Cheers
Example.jpg
Interestingly, the below screenshot is of a system setup in almost exactly the same way - fixed disks and with less free space on the data drive and yet this one backs up perfectly. Different version of server though - this is 2012 R2 and the problem machine is 2016. This makes me feel it isn't normal behavior for it to run the disk out of space then crash the VM if there isn't 'enough' disk space on the drive in question.
This is Windows backup - the built in backup that comes with the OS. We've used it for years just for the onsite backup and never usually get any issues with it (we also usually use a cloud based backup too although this particular customer declined that option).
Cheers
Example.jpg
This question needs an answer!
Become an EE member today
7 DAY FREE TRIALMembers can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
You USB stick size is smaller than your VM.
Now then, if D: is some other device, then maybe your backup software is using D: as temporary storage for some reason.
Either way, you'll likely have to increase the size of D: for your backup to work.