Could you have some snapshots taking the extra space?

 

What does df show us ?

here are a few snapshots.  If something more specific would be helpful please let me know.

Does your RAID controller support TRIM

can you post the output of ‘lvs’ please.

From what I can tell the raid controller (PERC H710 Mini) does not support trim.  I am not real sure how to definitively determine this.

google just informed me that the H710 does not support trim.  So, without trim, how would one release those unused blocks?  I am not sure if it matters or not but I am using SAS drives not SSDs

As it looks to me, you have overbooked the drives for your VMs (which is possible with Thin-LVM; you probably got warnings when doing so), and by calculating the percentages of used space, i come around that amount you described at first. So, it looks to me pretty accurate, that you have used that space. I personally would warn from oversizing volumes, because if they get fuller, it might fill up the thin volume completely, which causes possible damage (for example: if you would fill that vm-100-disk-1 with data, you would exceed that easily)

I respect the input, but i have to disagree.  Knowing what data exists and what I am storing its around 2TB.  With 6TB on Raid 5 being almost full its preposterous.  Yes, it's over allocated with the other VMs, but that is just laziness.  Home assistant has a 200 GB allocation, I think.  I knew it would not be close to that so i figured why not.  Irresponsible yes, but not the culprit, i think.

 

So, before this machine getting cantankerous and hard to deal with I am considering putting all of my plex data in a cloud somewhere as a backup, then backing up my other VMs and CTs.  Burn this whole thing to the ground just start over.  If I did so, is there something in initial setup that I should do different to mitigate this issue going forward?

In researching this further it appears that snapshots, and deleted data can fill-up a LVM - does not matter if SDD or HDD

And this needs to be reclaimed , if you do not have supported disks it cannot be reclaimed 

It’s a function of LVM

You could run

lsblk --discard

And see what values are returned

 

The future probably better using ZFS and new JBOD or SCSI HBA controller
 

 

I was afraid of an answer like that.  But that is ok.  Its the building I enjoy anyhow.  Much appreciated

It came back with 0 B trimmed.  I also tried running dd if=/dev/zero of=/zero bs=64k; sync; sync; sync; rm -f /zero on the ubuntu machine where the most data was.  Didnt seem to have any effect.

It is odd that Proxmox do state hardware RAID - maybe that’s supports trim/disgards

But many are using ZFS pools using a HBA or cards flashed with IT firmware not RAID

But ZFS is a rabbit hole for configuration with ZIL and Lharc SSDs in front of HDD !

I agree, I think my hardware setup + proxmox is just not a good setup.  I plan on flashing the perc H710 to IT mode and doing another install using ZFS and not Raid 5.  I think that is the only way to mitigate this issue in the future.  Thank you!