vSphere 5.1: Space reclamation with NFS datastores

We are running vSphere 5.1 using NFS datastores on NetApp. On Saturday, we removed/deleted a 150GB disk (thin provisioned) from one of our VMs. We backed up all the data on the disk -- about 100 GB. We took notes before and after the move.

The datastore volume has 540 GB capacity.

Provisioned space before deletion: 594 GB. After deletion, 436 GB.
Free space before deletion: 232 GB. After deletion, 235 GB.

I always thought that with NFS datastrores, space reclamation was done automatically. Why are we not seeing the free space go up?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ganesh Kumar ASr Infrastructure SpecialistCommented:
If you delete a file on a VMFS Datastore, the file system marks the affected blocks as unoccupied and reusable.
Unfortunately this information is not passed on to the storage system which believes that the free blocks are actually still in use.

With vSphere 5 VMWare introduced a new VAAI primitive called Thin provisioning block space reclamation (UNMAP).
This VAAI primitive should ensure that the storage system is informed if something is deleted and that the capacity is automatically released.

However, performance problems occured as these operations caused high storage IO and high latency.
VMWare has therefore recommended to disable the feature (actually it is disabled by default).

With ESXi Update 1 the command line tool “vmkfstools” was enhanced with an unmap parameter.
So it is now possible to release unused space manually as needed (but make sure that this is also supported from your storage vendor!)
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
How are you provisioning your datastores on your NetApp?

Are you using Virtual Storage Console ?

Are you using the VAAI plugins for NFS ?

or see here

pzozulkaAuthor Commented:
The datastores are thin provisioned on the NetApp. We are using Virtual Storage Console, but not VAAI plugins.

Also, for each Host in vCenter's Configuration tab, under Storage, all datastores are listed as Type = NFS, and only the local ESXi disk is listed as VMFS.
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It's recommended to deploy the VAAI plugins, and configure your hosts as recommended for NFS
pzozulkaAuthor Commented:
Her'es all I see when I run esxcli storage vmfs extent list
~ # esxcli storage vmfs extent list
Volume Name            VMFS UUID                            Extent Number  Device Name                           Partition
---------------------  -----------------------------------  -------------  ------------------------------------  ---------
i1whlesx02:datastore1  5270c817-4a522f52-083c-000af72f52a8              0  naa.6b8ca3a0f6d1f60019cd00c6124da7ca          4

Open in new window

As you can see, the only thing showing up is the local datastore (local disks on the esx host). None of the NetApp datastores are showing because they are all NFS. So I don't think the VMFS suggestions above apply.
Deploy VAAI and deduplication. It will save lots of headaches.
So small deletion most likely comes from deduplication - dupe data stays on disk to feed other copies.
pzozulkaAuthor Commented:
Thanks, I'll deploy VAAI. We already use deduplication. Not sure how it applies to reclaiming space from a deleted VMDK on an NFS datastore.
If there is deduplication enabled on volume deleting file will free less than that because of heap of zero blocks and heap of OS files shared with other VMs. Hope that explains where space disappeared.
With VAAI eager zeroed volumes are allocated in an instant, and VM cloning too. And it is VMWARE recommendation to use it. v1.0.21 is the latest one, and you can deploy it via VUM at regulat update cycle.
pzozulkaAuthor Commented:
Hmm...not quite. I can't say the space really disappeared. It just feels like it is not being reclamed after deleting the VMDK. See below.

BeforeBeforeAfter After
It sounds like the deleted disk was caught in a NetApp snapshot. The free space will only be reclaimed if all Netapp snapshots that were created before deleting the disk have been deleted/expired.

Use NetApp system manager and check for snapshots on the volume. Delete snapshots if necessary.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.