VMWare Datastore not accurately reflecting Free Space in regards to Snapshots.

Our backup takes what appear to be custom snapshots that do not get listed within Snapshot Manager. One of our clients went down Sunday and while troubleshooting it I found that there was 1TB of free space listed on the Datastore, but in reality 3 of the VMs had become stunned due to running out of space completely. There were 46 snapshots in the folder for one of the VMs. Though I resolved this issue by creating a 0 space snapshot and then running a consolidation, I'm STILL left with the question:

VMWare Version 5.5.0
Why is the Datastore not accurately reflecting Free Space in regards to snapshots?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mr TorturSystem EngineerCommented:
I think I never saw a thing like that in VMware, so maybe your issue is more at the Storage (San?) side, maybe you use San snapshots...?
In vSphere when you browse datastore you will see that there is a provisionned size and a real allocated size when you have VMware snapshots on your VM. But still, the max size of a snapshot file is not provisionned in advance by the system.
VMware snapshots are always growing with time AFAIK.

Before seeing this erroneous size, did you refresh your storage view in vCenter in the storage view tab?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Datastore space should be correct.

But a few pointers the datastore should be at least 20-30% free to allow for snapshots which backup apps use.

Secondly 46 snapshots also means 46 days whilst a VM is running on a snapshot performance of a VM will be poor.

VMware admins need to check daily for left over snapshots.

Otherwise all these snapshots will cause a deterioration in performance of the datastore.
DSTechTeamAuthor Commented:
It was running slow, yes. That much is clear but this is off topic.

These snapshots are generated by the backup program acting erroneously as the VMWare server was telling the program that it could not Quiesce but was actually creating snapshots successfully anyway. However that is not the point of this post.

@Mr Tortur
Storage is directly located in the server. Yes I did to a refresh and we also have an external monitor in Labtech that did also not correctly report the size as being completely full.

The base size is about 50% of the capacity of the Datastore's total size: 2.18 TB total and 1.08TB free (according to the Datastore's read.) Ultimately 50% of the space was basically taken up by snapshots of a single VM on the server but never reflected - which is crazy.

Problem is that when it quit on me on Sunday it had reported nothing wrong but had 46 snapshots that were created as a result of another issue but they in fact did live on the Datastore and were not being accounted for. One created each day for the last month and a half, and the Storage never reported all that space being taken.

Is that Free Space number only accounting for space not being Virtual Drive provisioned - not Snapshots? Because that's basically how it is acting. Essentially I return to my main question of

Why is the Datastore not accurately reflecting Free Space in regards to snapshots?
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Datastore space should be correct, sometimes however the vCenter Server inventory does not update correctly, requiring a refresh, or restart.

All provisioned space, should be reported correctly. (Server virtual disks and snapshots as well)

Also is this local storage of shared storage.

Without screenshots, it is difficult to diagnose.

Are you running latest versions of VMware vSphere 5.5? (vCenter Server and Host OS)
DSTechTeamAuthor Commented:
So bar the refresh which did not change anything I'm to believe that VMWare is reliable in reporting (which is serious) for less than 105 days? (Which is the current uptime.)

Nope, just local storage.

I'm sorry I was unclear:
This a server on it's own - dedicated local storage.
VMWare ESXi 5.5.0 2068190.

And I fail to see how it should matter at all but 2008 R2.

I do not have screenshots of when it was full completely, unfortunately as my concern was getting the company back online. My focus is now to figure out WHY the space changes were not reflected and as far as I know Labtech Virtualization Manager does a refresh every time it accesses the VMWare server on the storage to see the reflected size changes.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Is the datastore space being reported correctly now ?

Labtech Virtualization Manager probably only reads the information offered by vCenter Server.

Are vCenter Server and Labtech are consistent.

Your version of ESXi 5.5, is U2, not certain of vCenter Server, as you do not mention, but that was released in September 2014, there have been 10 releases since then, so it's old and not up to date.

I would recommend, a more update version, if this is Production.
DSTechTeamAuthor Commented:
It does look like it's reporting correctly now. I'm wondering if it wasn't refreshing automatically, even though it looked like it had.

I suppose we can call it for an update at this point for the client - do you know if there was a Storage refresh issue fix since any of the releases off-hand?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
There have been many many issues fixed with hosts and vCenter Server, over the last 15 months, and there have been updates to the Inventory Service.
DSTechTeamAuthor Commented:
I found the cause.

The Storage refresh never occurs - UNLESS you have a vCenter server, which forces the refresh. Anything without a vCenter does not auto-refresh.

This is pretty critical to know. I will not be forgetting this anytime soon. I appreciate your help Andrew, and will try to get that host updated as a matter of course regardless.
Paul SolovyovskySenior IT AdvisorCommented:
What type of storage are you using?  SAN/NAS snapshots may use the space allocated for the VMs, especially if you don't manage them and/or delete VMs that are still being held by the snapshot and not showing up on the datastore but are still part of LUN/Volume
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
So you are not using or have vCenter Server to configure Alerts on datastore usage ?
Mr TorturSystem EngineerCommented:
ok, well, good to know.
So if I summarize, you don't have a vcenter, only a standlone host, and when you refresh it manually it is displaying the good space stats, but it does not make auto-refresh on its own.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.