Link to home
Start Free TrialLog in
Avatar of sglee
sglee

asked on

Hyper-V running out of space while trying to create a snapshot.

User generated imageUser generated imageUser generated imageHi,User generated imageUser generated imageUser generated image
 I accidently created a snapshot on the VM "FT1" and there are two problems:
(1) Every so many seconds, when I open this VM in Hyper-V manager, it goes to "pause" state showing the screen in shaded color. When I start it, it goes back to shaded/pause  state.
(2) Now I am getting "Out of Disk" message on my Hyper-V server.

I did not intend to create a snapshot at all. I was expecting a snapshot manager screen as I chose "snapshot". But I think it is trying to create a snapshot and the server ran out of space. It seems that the logical thing to do is to cancel "snapshot creation process" because there is no hard drive space for the snapshot to be created in the first place, delete the temporary file "FT1_2481848F-1B1E-4667-A1A2-985A4E532138.avhdx"(18.9GB), and restart the "FT1" VM.

What should I do?

** I forgot to mention that I was trying to shut down the VM, but that is even stuck.
Avatar of sglee
sglee

ASKER

To Moderator,

 I have not seen any response so far. Did I select the right category?

Thanks
Avatar of David Johnson, CD
remove-vmsnapshot -name vmname

Open in new window

or in dire situation
get-vm | remove-vmsnaphot

Open in new window


Critical/Paused VM's can be turned off
Avatar of sglee

ASKER

@David
So when I run remove-vmsnapshot -name FT1_2481848F-1B1E-4667-A1A2-985A4E532138.avhdx in CMD, not only it will delete .avhdx file, it will stop the snapshot making process?
Avatar of sglee

ASKER

I ran the command:
PS C:\Users\Administrator> remove-vmsnapshot -name "D:\Hyper-V\Virtual Hard Disks\FT1_2481848F-1B1E-4667-A1A2-985A4E532138.avhdx"

and I get the following prompt:

cmdlet Remove-VMSnapshot at command pipeline position 1
Supply values for the following parameters:
VMName[0]:
Avatar of sglee

ASKER

I googled on "hard drive running out of space  read some articles  and so far, what I read, they all deal with the situation where snapshot file is completely created. At that point, you can delete the snapshot from HyperV manager.
However, in my case, snapshot has not been created and the hard drive has run out of space.
get-help remove-vmsnapshot

Open in new window



if powershell is prompting for something then wouldn't it be prudent to supply what it is needed?
It wants the vmname so type in the vmname as it shows in vm manager
Avatar of sglee

ASKER

So is removing temporary file the best course of action?
Won’t  hyperv Server trying to resume creating a new snapshot?
Avatar of sglee

ASKER

Moderator,
 
 I appreciate the effort of @David.
 However,  I like to work with another expert on this case. The reasons is that I am not getting answers from current expert in a timely manner in order to find the working solution.
That .AVHDX file is 18GB in size. That's a lot of change in the few short minutes since the snapshot being taken.

What is the storage layout on the host server?

The .AVHDX file can be mounted to access all "current" data in it.

The only way out of this given the fact that the partition the parent VHDX file is resting on is full is one of two:

1: Manually merge the .AVHDX file back into the parent. Note that this would require the two files to be copied to a partition that allows for a new parent file to be created which based on the Snip is ~420GB + ~420GB + 18GB for the .AVHDX.

2: Mount both parent and differencing disk and copy the data from differencing to parent. Copy the .AVHDX off. Delete the original .AVHDX. Boot/Start the VM.

1 is the safest route while 2 is risky. The backups are known good?
Avatar of sglee

ASKER

User generated image@Philip
(1) What is the storage layout on the host server? -->  C drive has W2012 OS.  D drive contains VMs.  F and G are external USB hard drives.

(2) The .AVHDX file can be mounted to access all "current" data in it.  --> I DIDN'T want to create a snapshot for FT1 VM. It happened by mistake. By clicking "Snapshot", I thought I would see Snapshot Manager like in VMWare, but unfortunately it started to create unwanted/unnecessary snapshot. The original FT1.VHDX was healthy and nothing was wrong.

(3) Can I somehow tell Hyper-V to stop creating this snapshot file (no space in HD currently and I don't need to create one)?
Do the properties for the FT1 VM show the .AVDHX or .VHDX file for its virtual disk?

What is the VM doing that it has such a high rate of change in a little over an hour?

Are the backups known good?

I'd set the parent .VHDX file as the virtual disk for FT1 then cut and paste that .AVHDX file onto one of the external drives to free up the space to get things going again. Any changed data would be had in the .AVHDX file anyway as mentioned it can be mounted to access data in it.
Avatar of sglee

ASKER

User generated image(1) Does the properties for the FT1 VM show the .AVDHX or .VHDX file for its virtual disk? --> .AVDHX
(2) What is the VM doing that it has such a high rate of change in a little over an hour?
 -->  I was removing some programs from FT1 VM and rebooted a few times in the midst. It also has ACCESS database which is constantly updated by users all day.
(3) Are the backups known good? --> I need to investigate.
(4) I'd set the parent .VHDX file as the virtual disk for FT1 --> I think I can do that, but I think I need to "shutdown" this VM to do that. Right?
(5) then cut and paste that .AVHDX file onto one of the external drives to free up the space to get things going again. ---> I can do that.
(6)  Any changed data would be had in the .AVHDX file anyway as mentioned it can be mounted to access data in it. --> There maybe some records updated here and there, but users can re-enter them easily.
ASKER CERTIFIED SOLUTION
Avatar of Philip Elder
Philip Elder
Flag of Canada image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of sglee

ASKER

(1) Right click the VM and turn it off. Then flip the disk to the parent VHDX (2) change the vCPU count to at least two
--> I can do that. About 1 hour ago, I started copying FT1.VHDX file to an external USB hard drive just in case. It is at 45% and I think it will be finished in two hours. Then I will try those two things you mentioned.
I will report back as soon as copying is finished.
Avatar of sglee

ASKER

User generated imageUser generated imageUser generated imageUser generated image@Philip
(1) Both FT1.vhdx and FT1_2481848F-1B1E-4667-A1A2-985A4E532138.avhdx have been backed up to external USB hard drive and D drive now has 25GB free space.
(2) I changed Hard Drive to FT1.vhdx and Processor to 2 virtual processors.
(3)  I am ready to start FT1 VM, Before starting it,  do I need to do anything with what is still in Snapshots section (in red box)? As you know, incomplete snapshot file is no longer on D drive.
Avatar of sglee

ASKER

User generated imageUser generated imageUser generated imageHaving confirmed that there was 25GB of free space on D drive, I started FT1. It came up with Safe mode/Safe mode networking/Normal mode ... etc. I choose Safe mode.
After scrolling some DLL files, it went shaded again. At that time, I checked D drive space and it was down to 82MB.
Then I realized that there is BIN file in D:\Hyper-V\Snapshots\E35C23F9-828B-458F-BA65-8386CD9E1CED folder with 13GB in size.
I am going to turn off the FT1 VM, backup  this BIN folder to the external USB hard drive, delete this file and restart FT1 VM.
I will report back.
Avatar of sglee

ASKER

@Philip,
 
 I am going to hold off starting FT1 VM again until I hear from you to discuss where we go from here.
 The reason is that even though I freed up hard drive space (25GB) by moving FT1_2481848F..avhdx to external USB HD, as soon as I started FT1 VM,  that space was down to 82MB. I don't know what just ate up that space that quick.
 Maybe we should remove Snapshot completely before starting FT1 VM again?
 Anyway I like to wait until we get to discuss.
 If I don't hear from you tonight EST, then we will discuss tomorrow AM.
Is there another .AVHDX file being created when the VM gets started?
Avatar of sglee

ASKER

@philip
I will be in front of the computer in the next 3-4 hours. Would you be available then?
Avatar of sglee

ASKER

User generated imageUser generated imageUser generated imageUser generated image@Philip,

 I discovered that $RECYCLED.BIN has three folders with different dates, however they all had the same file - FT1_xxx.AVHDX  that I deleted.
 Can I go ahead and delete these folders?
Avatar of sglee

ASKER

I checked the D drive space usage by major folders - Snapshots, Virtual Hard Disks and Virtual Machines
(1)
D:\Hyper-V\Snapshots has one folder with the size of 13.2GB:
 E35C23F9-828B-458F-BA65-8386CD9E1CED.bin is 13GB
E35C23F9-828B-458F-BA65-8386CD9E1CED.vsv 10MB
(2)
D:\Hyper-V\Virtual Hard Disks has 4 vhdx files for 4 VMs: 436GB
FT1.vhdx (419GB),
XP1 (20GB),  
XP2 (13GB),
XP3 (4GB)
(3)
D:\Hyper-V\Virtual Machines: 27.4GB. The breakdown of this folder is as follows:

D:\Hyper-V\Virtual Machines\3C42CC16-1DF4-4926-B73C-DCDA5C7C572C (2GB) .BIn, .VSV files
D:\Hyper-V\Virtual Machines\80DD39C2-7E9C-43BB-B306-D26245601821(Empty)
D:\Hyper-V\Virtual Machines\902E887F-E764-46CE-B379-0B62E821BB52 (Empty)
D:\Hyper-V\Virtual Machines\CA4375D2-EB65-46C3-85A5-49B83C8EDDB1 (26GB) .BIn, .VSV files
D:\Hyper-V\Virtual Machines\DDB0D718-BCBA-43D5-8668-3696AEB1B394 (20MB)

Is it typical to see 26GB size folder in Virtual Machines folder?
Also somehow I need to remove 13GB file in Snapshots folder.
Avatar of sglee

ASKER

User generated imageIt is a long story, but I managed to start FT1 VM and all looks good in it.
Since FT1 VM hard drive is now using FT1.vhdx (not .AVHDX file ) , can we remove what is in Snapshots section?
If the .AVHDX file is backed up then empty the Recycle Bin.

Any update on the backup story?

Leave the VM OFF.

Set up a new VM and use the option to choose existing storage. Choose the parent VHDX file. Set up the vRAM and vCPU settings along with the vNIC the same.

Boot the VM. Verify and set the network configuration then reboot.

If that works, then delete the original VM in Hyper-V Manager.

Any newer needed data would be found in the .AVHDX file on the external drive.
Avatar of sglee

ASKER

User generated imageUser generated image(1) Having deleted RECYCLED folder, D drive has 20GB.
(2) D:\Hyper-V\Virtual Machines\CA4375D2-EB65-46C3-85A5-49B83C8EDDB1 folder has CA4375D2-EB65-46C3-85A5-49B83C8EDDB1.bin file 26GB in size. Is this normal? Does this have something to do with  snapshot?
(3) Any update on the backup story? --> I have a complete backup of every single file in FT1 VM in an external USB HD.
(4) Set up a new VM --> I see what you are trying to do here. I think I may want to do that next weekend. At this point, FT1 is up and running and all workstation PCs can access files and folders from the server. So I like to keep it as it is, since Monday is tomorrow.

Can you shed some light on (2)?
Avatar of sglee

ASKER

User generated imageI rebooted Hyper-v server and all VMs are running like they used to and D drive has 18GB space left. We are good for now.
"Set up a new VM and use the option to choose existing storage. Choose the parent VHDX file. Set up the vRAM and vCPU settings along with the vNIC the same." --> I thought about this now and this makes sense. Since new virtual machine will be using existing good FT1.VHDX file as hard drive, no more drive space/additional RAM or CPU would be required as long as I turn off current FT1 VM, right?
Answer to the .BIN question: Yes, this is normal. Hyper-V lays down a file that is the equivalent to the amount of virtual memory assigned to each VM. One needs to keep this in mind when planning storage.
And yes, the current VM FT1 would not consume any resources when off. So, the "new" version would be the one to do that.
Avatar of sglee

ASKER

I will try to create a new VM for FT1 (and delete FT1 VM) in the next few days as it sounds safe and simple.
I will report back.
Hang on, but since the Hyper-V reboot the VMs are now running normally. Does that include FT1? If it does, then there's no point making any further changes?
Avatar of sglee

ASKER

Yes FT1 is up and running. However it has a trace of failed snapshot (as seen in the hyper/v manager screenshot) and incomplete avhdx file is not even in the D drive as I moved  that file into external USB HD.
So it is a kind of misinformation and nuisance. If I can somehow clear that, then I do not need to create a new VM.
This should show it:
Get-VMSnapshot -VMName FT1

Open in new window

This should delete it:
Get-VMSnapshot -VMName FT1 | Remove-VMSnapshot

Open in new window

Avatar of sglee

ASKER

I can delete the Snapshot while FT1 VM is turned on?
I don't see why not. It's just a listing at this point with no actual data associated with it.

Was there any data in the .AVHDX file that needed to be copied back?
Avatar of sglee

ASKER

"Was there any data in the .AVHDX file that needed to be copied back?" --> No. FT1.vhdx had everything up to the point & time when I, by mistake, clicked Snapshot option.
Good to hear.
Avatar of sglee

ASKER

User generated imageFyi, I thought, if I click "Snapshot", it would take me to some type of snapshot manager like in VMWare vShpere Client program.
I did not need to create a snapshot for any reason and I learned my lesson a hard way.
Avatar of sglee

ASKER

User generated imageAfter pressing [Enter] key after "get-vmsnapshot -vmname FT1 | remove-vmsnapshot", the cursor is just blinking.
I will try this again after hours.
Status?
Avatar of sglee

ASKER

The cursor is still blinking at the same spot.
Let me wait until everyone is done for the day, re-open Powershell and try to run Get-VMSnapshot -VMName FT1 | Remove-VMSnapshot.
I will report back this evening.
Avatar of sglee

ASKER

Philip,
Since all VMs are running well, I am not going to bother with erasing snapshot for FT1. The reason is that since I have moved avhdx file to external usb HD, it is not going to find the file that it will try to delete.
I just performed the same test in my Test W2016 Hyper-v server where I created a snapshot(they call it checkpoint now), moved avhdx file to another folder and tried to delete the checkpoint. It displayed an error complaining about avhdx file not found.
So at this point, I like to wrap up this case and I like to thank you for your help.
Avatar of sglee

ASKER

Thanks Philip for your help and I really appreciate it, particularly, over the weekend!
No worries, glad to help.