Datastore is 0 percent and all VM's shutdown.

Do you have another DATASTORE available to move a VM to?

MO

If you don't, you can take a USB external drive and create a datastore on it. Then you can migrate the storage of a VM (presumably one a light weight one that doesn't require a lot of disk I/O to operate) to the USB based datastore. Then you can turn all your VMs on and figure what you need to do to add more capacity to the existing datastore or create a new production one all together.

MO

Check for Snapshots.
https://www.experts-exchange.com/articles/10300/HOW-TO-VMware-Snapshots-Be-Patient.html

If you have another store with enough space, move  some of the VMDKs to another store

Michael,

If I delete stuff off the VM server, how to do I reclaim that space back to the datastore?

Michael,

I only have one Datastore and do not know how to add an additional data store.

Follow this guide to add a data store
http://howtonetworking.com/virtualization/esxi55-1.htm

For some reason when I try to connect to my vCenter Server.

I'm getting error: vSphere Client could not connect to xxx.xxx.xxx.xxx.
An unknown connection error occurred (The request failed because of a connection failure.  (Unable to connect to the remote server).

And I don't see a Disk/LUN listed when I try to add storage.

Check the below link
https://www.experts-exchange.com/questions/28992044/Vsphere-Client-could-not-connect-to-esxi-an-unknown-connection-error-occurred.html

I'm pretty new at vCenter and my Datastore is at 0% available space and all my VM's have shutdown.

It's possible that ALL your VMs have snapshots, and the snapshot can no longer write to the datastore because it has no space.

can you confirm this ?

USB External hard disks are not supported for using as a datastore.

I think we can ignore the vCenter Server issue at present, connect directly to the ESXi server, and confirm the snapshot situation.

You cannot just add an additional datastore, if you do not have storage available to do this!

Let me work with you to get this issue under control.

So, can you give me a screenshot of your Datastore.

Andrew,

There are no snap shots present.

Have you checked the VMs ?

as per my EE Article

HOW TO: VMware Snapshots :- Be Patient

you've checked the disk settings, and not just looked at the Snapshot Manager ?

Is the datastore actually present ?

What is the server, make and model, Storage Datastore size ?

Yes, I did check the disk setting and didn't see any that had the extra zero's appearing.

Okay, so if we assume zero snapshots. Then the datastore may have failed or disappeared.

So how is the server configured ?

Could you screenshot the datastore summary.

Data Store Summary

Data Store Summary
DatastoreSummary.png

okay so there is datastore called datastore1, 1.64TB I cannot see the capacity....

and if you hit Power On ?

Try this Summary
DatastoreSummary.png

I powered it on, it finally came on, but only have 7.97 GB available.

Okay.....there is another issue here.

For every VM you have, when it's powered ON, it also creates a SWAP FILE on the Datastore equal to memory in the VM!

So you MUST ensure when creating Virtual Machines, you keep at least 20% of your datastore FREE for Snapshots  and SWAP files.

You CANNOT use ALL the datastore space for virtual disks, because otherwise you will NOT be able to Power ON VMs.

So with 8GB left on the datastore, that is TOO LOW.

So how many VMs, how large are the virtual machine disks, and does that exceed 1.6TB ?

I do have VM with extra hard drive space.
Can I shrink the hard drive in order to give space back to the datastore?

Can I shrink the hard drive in order to give space back to the datastore?

Yes and no.

There is no vSphere function to shrink a VM, only grow it.

using a free application called "VMware Converter" it can be used to create a new VM with a reduced disk.

I have an article here I wrote...

Part 13: HOW TO: Shrink or Reduce a VMware Virtual Machine Disk (VMDK) using VMware vCenter Converter Standalone v5.5.2

HOW TO:  P2V, V2V for FREE - VMware vCenter Converter Standalone 6.1

now normally you would do this, before you ran out of datastore.....so you will have to

1. Convert or Export the VM to a workstation.
2. Destroy the VM.
3. Convert back, with reduced disk size.

To be honest with you, I would always create virtual machines with small disks, and grow them as and when they are needed, rather than creating and oversizing diisks....because expansion is easy, reduction/shrinking is awkward.

This was done by our third party vendor.

Can this be done while the users are on?

NJ-EWhite,

Easy way to get out of this is to add a USB external drive and create a datastore on it. I disagree with Andrew...you can create a datastore on an external USB drive. Here is a good guide: http://www.virten.net/2016/11/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-5/

It would just be used temporarily so you can generate some place to get your production environment back online. That's why I advised moving a VM that didn't see much activity or require much disk performance.

An alternative is that you can run down to your local computer outlet store and pick up an inexpensive NAS device. You can then connect that NAS to your vSphere host via NFS (provided the NAS supports NFS).

MO

Andrew,

I have a Mail Server that will not be using since we've have gone Outlook 365  online.

I will try and Export the VM to my workstation, convert it and add it back.

I will try to use your instructions to do so.

@Michael Ortega Disagree all you like it's not supported by VMware!

and states that in the article as well....

Please remember that using USB Datastores is not supported by VMware so if you have problems, do not open service requests for that.

Source
http://www.virten.net/2016/11/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-5/

It's a HACK!

Yes, running out disk space on your one and only datastore is also not a great condition to be in either. Extreme conditions sometimes requires some out of the box thinking. VMWare doesn't support datastores residing on a USB mass storage device, but I've implemented it in such cases as these without any problem. Also, it's just a temporary measure. In some cases, when I've had similar issues, VMWare support has assisted despite the fact that it included housing a datastore on a USB drive because they also realized the intent was not to run a production environment on it, but to deal with a temporary condition/issue. It helps to see things in "gray" sometimes, and not just black & white. I was simply providing the fastest, easiest solution to the problem. For shits and grins, I also provided a "supported" solution to the problem which was to go grab an inexpensive NAS at a local retailer and connect it to the vSphere host via NFS.

MO

@Andrew, I do apologize though. I read your original post as though it could not be done. You simply indicated that it was not supported by VMWare. I do agree with that. It's still the easiest and fastest solution to the problem. The risk level is extremely low.

MO

@NJ-EWhite,

Another option which you can do pretty easy if you have a space PC sitting around with enough local storage on it is flash it with FreeNAS (http://www.freenas.org/), create an NFS share to use as a datastore.

MO

How do I export the Mail Server from VMware?

I have an article for that as well...

Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE

same procedure for all versions of ESXi (4-6.5).

Andrew,

What is V2V and can this be done while the workers are logged in?

Virtual to Virtual conversion

I would recommend you complete out of hours, with no users accessing the server. What function does the server perform ?

see here

HOW TO: FAQ VMware P2V Troubleshooting

If you have good backups, why not just purge the mailserver from the datastore? Do you need the mailserver? Are you using AD Connect/DirSync with O365?

MO

Andrew,

Can I use VMware Converter, and convert existing server to workstation (and change size) while the server is powered down.

I don't have enough space in the datastore to power it up.

I thought you were going to Export/Archive a VM and delete it. A VM not in use.

So you have space?

So you cannot power on a VM now to perform a conversion?

VMware Converter can connect to ESXi or vCenter and perform a conversion

Question,  I've exported the Mail server to my desktop computer, I have a backup of it from last night and I also have also 3 replica of the Mail server.  I also installed the VMware vCenter Converter Standalone on to my desktop.  Can I connect to the ESXi or VCenter via vCenter Converter Standallone via my workstation?

You can but it's recommended to install on the VM to be converted to avoid network and permission issues

So the mail Server could be deleted

Yes, but if I delete it, how do I get it back if I need to retrieve stuff off of it?

Re-Import it, Restore from Backup.

I'm running the conversion as we speak.

So, if one of your primary means of restoring the VM (mailserver) in question is through the backup you already have, why are you going through all this trouble? Just purge mailserver from the production datastore. You can restore the VM when you've permanently addressed the storage capacity issue, but at least with this you'll have a running environment again NOW. Are you concerned that your backups aren't good?

I did advise doing this yesterday. Perhaps I just don't understand the gravity of the situation. All VMs being down just sounds critical to me, so I would have gone the path of least resistance. If backups are good and the mailserver VM really doesn't have a production role any longer, the risks are low...just seems clear to me what would need to be done to restore the environment.

Sorry that you're still working on this.

MO

Michael,

I managed to bring them back up the same day it went down, but with only 7.85 gigs of free  space.  I did manage to export the VM and Convert the VM now I'm going delete the VM.  

Now, how do I correctly delete the VM.   Do I just go to the Datastore Browser and just delete the folder?

I'm not quite sure how to delete a VM.

You can right-click on the VM in question and there should be a "Delete from disk" option.

MO

The VM that I converted to my desktop total storage is 109gb.

If I was to convert the vm from my desktop back to a VM, is that the total space that will be taken up by the datastore1?

Yes, plus whatever memory you allocate to it. Just make sure to THIN provision the disk(s).

MO

Well that depends, if you decide to RE-SIZE it!

and then it would be smaller!

Does your VM actually have 109GB on the partition, as seen from the OS, or is that just the disk size ?

I've since deleted the VM and am now going by the size of the VMDK file.  

They total to the 109gb.

Well you have the function to re-size the virtual machine disk, on conversion back to ESXi.

Converter is intelligent here, and it will not re-size the disk, beyond what data is actually in the vmdk!

So if you have 100GB of data, the minimum size of the VMDK would be approx 100GB, but if you only had 20GB of data in a 100GB VMDK, VMware Converter, could change the new disk between 20GB and 100GB or larger!

So if you are on the limits of what you currently have free, and I would suggest 20-22% free on the datastore, you could started grabbing some storage space back.

OK!  Thanks for all your help.

no problems.