We help IT Professionals succeed at work.

VMWare There is no more space for virtual disk

2,709 Views
Last Modified: 2016-04-06
Guys,
I'm getting this error.
And when I try to power off the machine it doesn't let me because of the same message.

I try to access via console and nothing it doesn't let me enter the console same error.

So what do I do?
Comment
Watch Question

Steve AgnewSr. Systems Engineer
CERTIFIED EXPERT

Commented:
Sounds like you have run out of physical space on a hard drive.. in the start menu type in disk cleanup and see if it can auto find you some space to clear.  I like to use Sequoia View to visually see what is taking up space on my hard drive to see what I can delete that is the largest.. but if you're outta space you may not be able to download/install it... http://download.cnet.com/SequoiaView/3000-2248_4-10549662.html

Author

Commented:
I was able to power off the VM but it won't let me power on, it comes back with the same error.
I do have around 4TB of space in the Datastore.

Please help!!!
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
We answered this before, and you deleted the question!

You have three snapshots associated with this virtual machine, when a virtual machine writes to a snapshot file, it grow and grows, and grows.... you get it.... until the datastore is FULL, and then the VM fails. Snapshots are missed because they are left by virtual machines, after backup applications, or you have forgotten, after using manual snapshots.

Also VMs run poorly, e.g. performance is poor when on a snapshot.

We can work with you, to get this issue solved...in the meantime, have a read of snapshots in my EE Article, and hopefully when I submit this time, the questions is not deleted!

see my EE Article

HOW TO: VMware Snapshots :- Be Patient

What I need from you, is to follow my instructions EXACTLY and do not stray of the path, and meddle, or mess around, otherwise corruption can occur, if not done already.

1. Do you have any other datastores ?

2. Do you have other VMs which share the same datastore ?

we need some screenshots of the datastore, storage space, and a screenshot of te error message, remember WE CANNOT see what you are seeing, we need to ask you!

screenshots would help.

@Steve .... this issue is occurring on the VMware vSphere ESXi host, not in the VM.

Author

Commented:
Yes, only one datastore
Yes, many VMs sharing the datastore

Author

Commented:
I have 3 snapshots for this VM.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
I KNOW you have 3 snapshots. (you deleted the other question! before I could respond!)

SCREENSHOTS PLEASE of datastore and storage space.

I need to see the storage space, left, with a single datastore, how much is FREE.

Author

Commented:
So, what do I do, please help!!!

Author

Commented:
Ok, schreenshots are coming
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
okay, good, let me have the screenshots, and I can advise. Be patient, I'll hold you hand, but please do not mess, or fiddle, and make the situation worse.

Author

Commented:
Now I'm waiting for your comments, let me know if you need more information.
Thanks.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
this is a little odd, because it does state 4.5TB free.

can you screenshot the error message, and also the settings of the VM Disk, in the VM.

So this is a vSAN datastore ?

what is the capacity of the datastore ?

it states, 40TB, but 75TB provisioned, are you using Thin Virtual Machine disks, and you've overprovisioned your datastore ?

and now this is a combination snapshot and thin provision.

Author

Commented:
The virtual disk in question is 9TB in size, and the users have used about 5TB.
Take a look at the Datastore contents and you will see the 9TB file there.
error.JPG

Author

Commented:
Yes this is a vSAN.
40TB in capacity.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
can we check the disk settings, to check it's a snapshot.

e.g. Right Click the VM Edit settings, and select the disk, and lets check,

are these thin disks ?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
okay, let me catch up and check the screenshots

Author

Commented:

Author

Commented:
Can I unmount the problematic virtual disk, and start the server?
I have a whole bunch of other information there that people need to access.

Author

Commented:
Andrew, any ideas?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
I have a whole bunch of other information there that people need to access.

I'm afraid at present you are DOWN, and you will need to advise them of that.

okay, we need to deal with the snapshots here....

now, snapshot merging (deleting) can take many seconds, minutes, hours or days, depending upon the size and speed of your datastore. You will need to be very patient.

1. do not cancel the operation.
2. do not reboot the host servers.
3. do not power on the VM.
4. do not restart networking agents
5. do not restart vCenter Server.
6. basically do not do anything other than walk away, grab a coffee, tea, red bull....

and BE PATIENT

it will possibly sit at 90, 95, 99% and look like it's hung BUT it will complete....

You need to free up space on the datastore.

1. there is a folder called isos, can we move it or delete it some where, move to local PC etc

is there any other folders we can delete or move, storing isos on vSAN is expensive usage of stoarge ?

Author

Commented:
Yes, I can delete the isos folder.

Author

Commented:
do I do that?

Author

Commented:
Andrew please help me out man, I'm having a heart attack here.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Click the isos folder, and remove all the files, right click and delete each file.

then remove the folder, again right click delete.

Author

Commented:
I just tried but it said that it can't delete the iso file.
isoerror.JPG
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
you will have to check, if it is connected to any VMs, and disconnect and then delete.

Author

Commented:
I just check and release the access to this iso file.
But I'm still can't delete it.
This is how the other VMs configuration looks like.
DVD.JPG

Author

Commented:
But why is it saying that it has 4.5TB of Free space, I don't understand.

Author

Commented:
I have a Virtual Disk in a different VM that we could remove, and that would free some space.
But you will need to guide me through that process.

Does this sound like a good alternative to free up some space?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Anything that can release space, will help.

Lets not worry about what things state, we need free-up space....to start the snapshot process.

Author

Commented:
Andrew,
I really need some other users to access information in other virtual disks I have on that server.
Is it an option to disable the virtual disk that is full, so that I can restart the server?

Author

Commented:
I can also see that in the image attached to this message the VM has reached the 17.19TB of space it has available.
Is it an option to grow the virtual disk in question so that it can let me start the VM and then I can delete a lot of files I know I can delete?

Author

Commented:
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
The snapshots need to be handled correctly, deleting files in he VM is not going to help the snapshot.

The snapshot MUST be dealt with.

I'm afraid all your disks on this server, have snapshots attached.

Even if you could remove a disk, to get it started it's only a matter of time, before it would fail again, because it could not write to disk because of the snapshot, and the danger is you could the  LOSE ALL or the data in the VM could become corrupted.

Did you delete the VM, and free up some space ? and we can proceed.

The quicker you free up space, the quicker we can start the snapshot merge.

Author

Commented:
What if I go to the snapshot manager and delete the snapshots from all the VMs from there, would that help?

Author

Commented:
In the Snapshot manager there is also an option that says consolidate.
Wouldn't that delete some of the snapshots?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Okay, please follow my instructions.

did you free up space on the datastore.

In the Snapshot manager there is also an option that says consolidate.
Wouldn't that delete some of the snapshots?

Possibly, I did post originally, do not stray off the path and follow my instructions, otherwise you will end up in a world, you do not want to be!

What if I go to the snapshot manager and delete the snapshots from all the VMs from there, would that help?

FREE UP SOME SPACE.

Did you delete the VM and free space ?

Author

Commented:
Andrew,
I just removed a virtual disk with a 1TB size.
How can we check if we have enough space now?
VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017
Commented:
This problem has been solved!
(Unlock this solution with a 7-day Free Trial)
UNLOCK SOLUTION

Author

Commented:
Question:
I went to the machine edit settings and click on removed the 1TB virtual disk that I was not using.
But when I go to the browse the data store it seems like the vmdk files are still there, should I remove those vmdk files by hand?
1TB-VDFREE.JPG

Author

Commented:
Also shouldn't I delete just one or 2 snapshots first?
I would like to have at least 1 snapshot left.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Delete the virtual machine disks manually, you probably did not select delete from disk.

Snapshot are NOT BACKUPS.

REMOVE THEM ALL. DELETE ALL.

What are you going to do with them ALL, rollback the server and lose all your data ?

Author

Commented:
Ok, I have never deleted vmdk files, so I need your help to pinpoint the ones that need to be deleted.
From the image attached to this message I think I have to delete the following files:

YYCDP-FS-01-2.vmdk
YYCDP-FS-01-2-000001.vmdk
YYCDP-FS-01-2-000002.vmdk

This is because the virtual drive deleted was 1TB in size and there are no other 1TB virtual drives in this server.

Am I correct in the file identification?
1TB-VDFREE.JPG
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
If that is the assumption you are making then those are the files that need deleting, the 0001 and 0002 are snapshots.

You don't use Snapshots are Backups do you because there seem to snapshots on this VM ?

this is very bad practice, I'm not surprised you are running out of storage space, and have used up 40TB.

Author

Commented:
Ok, I have deleted the vmdk files in question.
Now I have to go into the snapshots and delete them all, as per your instructions above.

Correct?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
That is correct DELETE ALL.

Make sure you select DELETE ALL.

Author

Commented:
Ok, DELETE ALL snapshots it is.
The process just began,
Wish me luck

Author

Commented:
It is at 25%

Author

Commented:
It is still at 25%.
How long does this normally take?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
now, snapshot merging (deleting) can take many seconds, minutes, hours or days, depending upon the size and speed of your datastore. You will need to be very patient.

But 25% is quite fast...

and 25% does not mean it's not doing anything, as the indicator is a bit rubbish.

Have you been manually snapshotting ALL your VMs ?

Author

Commented:
It is now at 26%
So much for the very fast.

But it is still going.

Author

Commented:
It is now at 50%
While this process completes, I will open a new case to discuss snapshots. Can you take that case?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Yes, see the Progress Bar is really rubbish, if it's jumping fro 25%, 26%, 50%...

If you post another question myself or other experts will be glad to assist.

Author

Commented:
It is still at 50%.
Should I also delete all the snapshots from the other VMs?

I just created a new question to discuss snapshots, please feel free to join in.

Author

Commented:
It is at 52%
You are right we don't know when this is going to end.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
You should really know how and when to use these, you are not the first VMware Admin, that's been caught by them, or the last.

We would recommend , not use them and backup first, and then apply changes.

You do have BACKUPS of your VMs ?

What do you use ?

If you are going to use them, Snapshot - Apply Patches - Patches Worked - Delete Snapshot.

This LONG delay is caused BY YOU, leaving the VM on a snapshot too long, and it's going to take this length of time, possibly to fix ALL your VMs.

and you've wasted lots of space, and huge overhead on your SAN, with all these snapshots.

and performance will be poor.

Author

Commented:
You are right, Mea Culpa.

It's at 55%
I just hope the server starts when this snapshot deleting finishes.
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
The merging operation is removing ALL the DELTA Snapshots from the vSAN SAN, and free-up the storage.

and you should do this WITH ALL YOUR VMs.

But this will allow the server to start normally.

Author

Commented:
Got stuck at 55%

Question:
If I close the vSphere client, will the process running in the server?
DELETESNAP55.JPG
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
it will appear to stick.

the process is completing in the back ground on the server, you don't need the client open.

Author

Commented:
Still at 55%
How do I know if it is still running or if it got stuck?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Be Patient.

Look at the files in the VM Folder.

Author

Commented:
Can I click "close" in the Snapshot Manager without affecting the process?
SNAPMANAGER.JPG
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Yes, you can click close.

the process is completing in the background on the server.

Very bad use of Snapshots!

Have you not noticed your servers getting slower.....

Author

Commented:
I just look into the datastore and it looks like it is processing the heavy heaters, snapshot files for the 5TB and 9TB virtual disks.

Some of the 00000X.vmdk files from the small virtual drives are gone, so I guess that was the supposed 55% of the process.

My estimation is that this is going to take at least all night.

Should I be able to see the size of any of these files going down during the process?
VMDKFILELIST.JPG
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
not really.

again, it's best to walk away, and not watch it!

Do you not have a Backup program for VMs ?

What do you use to Backup VMs ?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
I'm about to disappear to my bed, I'll be back later.

Author

Commented:
Andrew,
I'm going home, watching 55% not moving is torture.
I will connect remotely and I'll let you know if the process status changed.

Thanks.

Author

Commented:
It's at 76%
I check the files and it is now processing the last virtual drive "the big one" 9 TB

Question:
Once the process ends successfully, I can just try to power on the VM, correct?
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Question:
Once the process ends successfully, I can just try to power on the VM, correct?

Yes, you can power on, you will find, all the snapshots should have gone, and you've got much storage space back.

and remember you other VMs, could also be at risk of failure, because of the Snapshots.

Author

Commented:
Good Morning Andrew,
The server is back on, thanks for everything.
:-)

For the other servers, same procedure?

1. power off
2. Snapshot manager
3. Delete all
4. Be patient

Author

Commented:
Mr. Andrew,
I will take care of the snapshots in the other VMs.
I would probably gone crazy without your guidance by now, so thank you again for your patience.

Best Regards,
Cargex
Andrew Hancock (VMware vExpert PRO / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Hey I'm glad your back in business.

For the other servers, same procedure?
- almost

1. power off
2. Snapshot > Take Snapshot
2a. Wait 1-2 minutes
3. Snapshot Manager and Then - Delete all
4. Be patient

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions