Need to determine why is the extra disk space used on one of the two cluster pools which should be identical in disk usage.

Are there any snapshots that are chewing up the extra space on the cluster?

I asked the customer that question already, and they said that there are no snapshots. I'll be logging remotely to their clusters tomorrow to see what I could find. I also asked about the allocation unit size on disks on both cluster storage, and they said there were the same at 4096.

Need to verify what backup product is being used.

Hyper-V has some issues with volume shadow copy snapshots which is the service used to create the backup snapshots. It tends to leave orphaned .AVHDX files behind that were created during the backup snapshot which can take some time depending on storage capabilities, VM resource usage, and other factors. Once the snapshot is done and the backup has been started the backup service will try and delete the snapshot with the changes merged back into the parent .VHDX file. Somehow Hyper-V tells the backup service that the merge was successful when it actually wasn't.

The other thing to check is the configuration files. Some of them get bloated also because of the way Hyper-V and snapshot backup products interact.

Thanks Philip, I am looking for a way to discover those orphaned .avhdx files on a cluster. I see some scripts on the internet,  but I have tried them and they don't work. The target OS here is 2016 Server.

Okay, start with:

Get-VMSnapshot -ComputerName "Hyper-VNameHere" -VMName "VMNameHere"

Select allOpen in new window

An asterisk can be used to get all VM's snapshots. The orphans will be "Snapshot Type: Recovery".

To remove:

Get-VMSnapshot -ComputerName "Hyper-VNameHere" -VMName "VMNameHere" | Remove-VMSnapshot

Select allOpen in new window

There are times where we need to trigger a CheckPoint/SnapShot that will then trigger the needed orphans to be merged back in:

# TODO Create a CheckPoint then Delete it

Checkpoint-VM -VMName "VM0"
Get-VMCheckpoint -VMName "VM0"
Remove-VMCheckpoint -VMName "VM0"

Select allOpen in new window

A just in case:

# Merge the AVHDX - Requires VM to be OFFLINE
Merge-VHD -Path "C:\ClusterStorage\CSV01\VM0\Virtual Hard Disks\VM0_D0_6500000D-2FDA-4B90-9085-GUID.avhdx"

Select allOpen in new window

I am mostly interested in those orphaned files that are not going to be merged, but just deleted. Do you have any of those examples?

The orphaned .AVHDX files _would_ be merged. That's the key. They are still part of the VM's parent .VHDX file chain. One cannot just delete them thinking that's all that is needed. That would be real bad.

ok, so I would just need to discover the orphaned files, I wouldn't go on to merge them at this time, just a report on their existence. I would need to get this report on the Hyper-V cluster.

The PowerShell above does that.

I got this customer to run the scripts on all of the hyper-v hosts in the cluster, and he reported that no results came up.

Are we missing something here?

If nothing comes up then there's another culprit.

Have them look for *.vm* files which are settings files. They tend to get bloated over time. They will be in the \Virtual Machines\ folder(s) that usually reside with each VM.

*.vmcx

*.vmgs

*.vmrs

I checked with the customer, he told me that the VM files are total around 130 GB. The storage difference between the clusters is around 9TB, so this is too small, and has to be something else that's consuming the disk space on the cluster storage.

Is there a way to do file compare to determine which files / folders are different (extra) on those two clusters?

Yeah, this is too complicated a question for a Q&A Forum like this unfortunately.

One needs to have hands-on at the console to run PowerShell scripts to find the culprit as there can be many.