Hyper-V Shuts Down All VMS Due to Critically Low Drive Space

This is a huge problem for me!
This happened so  frequently on my old server  that I was motivated to build a hot swapable rack mount Hyper-V host with (2) 3TB RAID 5 Arrays (SystemRAID and StorageRAID). I know that's not that much these days but anyway I woke up to a catastrophe today! My DC is one of the VMS.

Sorry for the rant but what is the easiest way to deal with all of the left over junk that is hogging up so much space? I think that the VHDX's are suppose to re size after a restart but short of attempting to delete files with all guests running and depending on sharing violations to protect me from deleting a file that is in use, what is the preferred way because this issue just cannot continue?
A_AmericanELectricAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Lee W, MVPTechnology and Business Process AdvisorCommented:
The preferred way is to monitor the systems and never let the disk space get low enough to cause a problems.  You can also used fixed VHDs/VHDXs so they can never grow.

Partition the drives so that no one server can knock out all the others.  Partitions/logical drives can act as hard caps on growth for when you over-commit the VHD size.

Another thing you can do is use FSUTIL to create a 50-100 GB file instantly (or a set of large but smaller ones) so that you have something you can easily delete to resume functionality and give you a LITTLE time to make new accommodations (get larger drives, etc).

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
We use FIXED VHDX files. This is one method for avoiding overprovisioning.

We do NOT snapshot/checkpoint VMs. When that gets done a Differencing Disk gets created (AVHDX). That file keeps growing until the snapshot/checkpoint gets deleted and the parent VHDX and AVHDX file(s) get merged together. NOTE: This process creates a new file so enough space must be present for that file!

If you are out of space and need to merge differencing disks into the parent: Manually Merge .avhd to .vhd in Hyper-V - TechNet Articles - United States (English) - TechNet Wiki

EDIT:
 + Do NOT delete the differencing disks. This will kill the VM.
 + You will need to copy off the parent and each AVHDX file to another system for the merge process. A server with high performance RAID would be a good thing or a desktop with enough SSD storage to run it.
A_AmericanELectricAuthor Commented:
Thanks lee but can you give me a little more detail on FSUTIL?
Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

A_AmericanELectricAuthor Commented:
Thanks phillip. Yes I learned to stay away from chekpoints the hard way. This seems like its a huge hassle for many.
Seems like there could be more advance warnings. I'm running a few virtual servers and they hum along increasing in size all the time. I don't login to some of them for months!
Lee W, MVPTechnology and Business Process AdvisorCommented:
Reference http://ss64.com/nt/fsutil.html for examples and usage (or just type FSUTIL /? on a command line).

fsutil file createnew D:\spacefiller1.tmp 5120

The above command creates a temp file called "spacefiller1.tmp" of 5K (exactly 5K to a computer).  Run it from an administrative level command prompt.  And 5K is way too little for this... you probably want to add 6 zeros or more.  It only takes a second (less) regardless of file size.
A_AmericanELectricAuthor Commented:
Ok I got that but how does the space filler help?
Lee W, MVPTechnology and Business Process AdvisorCommented:
It can be a NIGHTMARE trying to free up space for a VM in a paused critical state.  If it gets that way, simply delete the space filler to instantly get room back and restore the VM.  Otherwise, especially if this is on your largest drive and you have NOTHING else to move/delete you are stuck for potentially hours.  It's an emergency buffer of space you can easily reclaim.  How much you allocate, how many files, etc, is up to you.  But having had that happen on one of my semi-test systems, I found this was a great, simple, easy way to ensure things can easily be brought back online QUICKLY.
A_AmericanELectricAuthor Commented:
AHHH I get it now0
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
Ideally one would not overprovision their storage in the first place. :)
Lee W, MVPTechnology and Business Process AdvisorCommented:
ABSOLUTELY AGREE.

Especially in production systems.  Test/lab systems, I'm far more inclined to not care too much...
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Hyper-V

From novice to tech pro — start learning today.