Provisioned space showing far more than capacity

Since upgrading our ESXI environment from 5.1 to 5.5(including building new VCenter servers), we have noticed that our "provisioned" storage on some of our datastores are now showing as double the size of the actual VMs on that datastore.

For example, In one of my datastores, I show a provisioned size of 1.65TB.  This datastore only has 1 VM on it with 2 disks.  The first disk is a Thin provisioned 50GB disk.  The second is a Thin provisioned 830GB disk.

We originally provisioned the datastore to be 1TB.  There are no active snapshots for this VM and his has been confirmed both in the datastore view as well as a visit to the command line using SSH.

In my open ticket with VMWare, the tech tried to tell me that we just over-provisioned our datastores but this is not true.  We know what size we provisioned them...VSphere is showing that size as different now and we would like to resolve this.

On the summary tab of the datastore, we see the following:

Capacity: 1.00TB
Provisioned Space: 1.65TB
Free Space: 216.37GB

There are no snapshots at all.  The VM has 4GB RAM.  It has a total of 880GB disk space(although all of it is not being used)

Any ideas while I wait for VMware?  Our ticket is still open while we do research.  Taking the VM offline(removing from inventory and re-adding) is not an option as it is a production VM(File Shares) and it is used all day every day.  Please help.
LVL 2
Jonathan BriteSystem AdminAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Is this local disk, NFS export, or iSCSI LUN ?
Jonathan BriteSystem AdminAuthor Commented:
This is an ISCSI LUN, sorry Andrew, I should have mentioned that to begin with.  Our virtual environment is all hosted on our Netapp FAS3240
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Jonathan BriteSystem AdminAuthor Commented:
Andrew,  Maybe I am missing something but I do not see how this applies to me?  We are not looking to reclaim disk space and our NetApp is showing that the storage pool/Lun is still configured the way we originally did it.(In terms of size/usage).  This appears to be a display thing in VSphere 5.5.  It is just showing that the datastore is provisioned at 1.65TB when it is not now, nor was it ever.  

If we look at the datastore itself, and add up all the files in all directories, it shows as accurate.  I am just irritated that VSphere is claiming it is provisioned at 1.65TB when it isn't.  Am I missing something here?

BTW, Thanks for your help.
Jonathan BriteSystem AdminAuthor Commented:
Something else worthy of note.  If I click on the Datastors and Datastore Clusters tab in Vsphere, and select one of the datastores in question, then select the virtual machines tab.  The "provisioned space" and "used space" is accurate.  However, it is not accurate on the summary tab, so vsphere thinks the datastore grew in provisioned size for some reason while the vm provisioned space size remained the same the entire time.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So, you have a 1TB datastore, which shows as 1,6TB used ?
Jonathan BriteSystem AdminAuthor Commented:
We have a 1TB datastore that is using 880GB or so, but it is showing as being provisioned to 1.6TB which simply is not the case.
robocatCommented:
According to VMWare, the "provisioned space" is:

Total allocated space that the virtual machine can commit up to. Includes vmdk, swap, snapshot, and other virtual machine files such as NVRAM, configuration files, and logs. This metric includes uncommitted space.

So the provisioned space for your datastore is the worst case for all of the above for all of your VMs.

E.g. : even if there are no snapshots now, the number includes the possibility of snapshots in the future. And it assumes the thin provisioned disks will be fully allocated.

Perhaps vShpere 5.5 has a different approach to calculating this. We have a 150GB datastore (for swap) that shows 1TB provisioned, so you probably shouldn't worry too much about this number.
Jonathan BriteSystem AdminAuthor Commented:
The problem I have with this is the following though.

Lets say we are have to snap this VM just before we start copying files to a new partition.(adding a new drive in a new datastore)  As we can not have down time, the new partition would get copies of files on the existing partition.(using something like Robocopy).  We would run an initial seed of everything we want moved, and then when we do the cut over(on a weekend ofcourse), the end users would not really be affected(aside from maybe a log off and back on again).  This is a very real scenario that we will have to do on this particular VM in the very near future.

As the physical capacity on this datastore is only 1TB, and 880GB of it is already in use, what happens when the snapshot size, and the normal vmdk, swap, memory, etc...exceeds the maximum physical capacity of the disk?  It will crash the VM from what I understand.

VSphere will let it keep going and going until it runs out of space.

Now I realize this is merely a reporting thing, and that it is our job to ensure that we have enough storage available on the datastore to do whatever we need to do and not run out of physical disk space, but we expect that our overall view of our datastores in VMware will give us an accurate picture of what we have provisioned, what we have used, and what our actual capacity is.

As I stated before, this only started happening in vsphere 5.5.  It was reporting fine in vcenter & esxci 5.0u3 and only stared happening in our new vsphere and esxi 5.5u2.
Jonathan BriteSystem AdminAuthor Commented:
"So the provisioned space for your datastore is the worst case for all of the above for all of your VMs."

If this were actually the case, would I not see the same issue across all my datastores?  I am only seeing it on 3 datastores out of 20.  We use Thin provisioning everywhere and we do not over-provision anything.

In other words, if we make a 100GB Thin provisioned drive, we make sure there is 100GB free space on that datastore.   So while we do use thin provisioning, we do not over-provision anything here at all.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It will crash the VM from what I understand.

Correct. Why are you using Snapshots for what purpose ? You have to make certain when using snapshots you do not run out of disk space, and if you do, you could end of corrupted your VM disk.

Also performance is an issue when a VM is writing to a snapshot disk.

see my EE Article

HOW TO: VMware Snapshots :- Be Patient

We make sure ALL our datastores have 20-30% FREE at all times.
Jonathan BriteSystem AdminAuthor Commented:
Generally, when we create our LUNs, we increase the size to 20% more than we actually need.  So on the SAN, if we wanted a 1TB datastore, we would make it 1.2.  In Vsphere, we will try to only go to no more than 800GB usually.  So we generally have 20% free space on all datastores all the time.(This usually accounts for maximum filled disks as well).  In the case of Thin provisioning, we make sure that our datastore has the physical capacity to hold the entire set of disks assuming they are all full.  In this case, we are only using a few hundred GB though, not the full 850 or so.

As far as Snaps, we use Veeam B&R 8.  We do daily snaps of most of our VMs for backup purposes, they are then removed immediately, so no snaps are ever more than a few hours old(depending on how long the backup takes).

As a general rule of thumb though, when we are about to make any significant changes to a VM, we snap it first, so we have something to fall back on should something go wrong.  This also applies to patching our VMs as well.

It's my understanding we are doing everything best practice.  We are pretty consistent across the board with everything we do, and that is why we do not accept VMwares response that we simply "over-provisioned" the VMs or that Provisioned space means:

"Total allocated space that the virtual machine can commit up to. Includes vmdk, swap, snapshot, and other virtual machine files such as NVRAM, configuration files, and logs. This metric includes uncommitted space."

If this were the case, we would see this same behavior across all our datastores would we not?  Instead we are only seeing this behavior on 3 of 20.

Thanks for helping me all btw, maybe I am being dense here(not the first time), but I expect to see consistent behavior across our entire virtual environment because we are so consistent in how we do things.

Also Andrew, I read your article a long time ago and it is the reason why we now get alerts to any snapshots more than 4 hours old.  We know that snapshots are not backups, and I always correct people who think they are.

Great article by the way, I am sure it helped many folks get a better understanding of snapshots.
robocatCommented:
I would not obsess about this provisioned space metric, because the number is pretty useless. The vCenter web client doesn't even show this number any more.

It is much more interesting to use some tool for free space monitoring and alerting. Even more interesting is a tool that does trend analysis and predicts when you will run out of free space in the future.

E.g. take a look at vCOps (now called vRealize (?) I wonder if it is still free) which will do these things.
Jonathan BriteSystem AdminAuthor Commented:
We currently use VeeamOne for these purposes.  I still have the ticket open and want to know why it changed from 5.0 to 5.5.  I just want to make sure this wil not cause any issues later on down the line.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We have seen issues with Datastore calculations with vSphere, and it often gets reported here on EE, despite Admins going off, and searching all the VMFS datastore, the numbers do not add up.

@robocat vCOps it's Foundation which is included with your license,  or FREE!

If you have purchased a Suite license, it's also included.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jonathan BriteSystem AdminAuthor Commented:
The best answer I can get.  Thanks as usual Andrew and keep writing your papers.  Many of us read them religiously.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
@Jonathan Many Thanks for your kind words, I'll keep on writing, I just need to find some time...but nice words of encouragement!

Glad you like them....

Andy
Brubaker6Commented:
I know this thread is old, but I have just run across this same situation.  It only happens on one of my datastores.  .  I thought it was addressed in the following patch but it seems like it was meant for the reverse (where the VM is showing inaccurate amounts, not the datastore).
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2110199

I am in the process of apply patches to see if any of them fixes the issue.  Are yours still reporting incorrectly?  I only have 1 VM on the datastore.  I am wondering if I move it to a different datastore and refresh if the current datastore will reflect the correct amounts?  Will the new one show overprovisioned?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.