• Status: Solved
  • Priority: High
  • Security: Public
  • Views: 169
  • Last Modified:

ESXi no more space for Virtual disk

My VM wont boot up because it is saying no more space for Virtual disk TSERVER-000003.vmdk.  This is a windows server 2012r2.  -000003  is the hard disk 1.

How can I get this VM Back up and running.  I do have another local datastore.  If it is possible to move just that virtual disk??

Also, are there any files in the datastore that I can safely delete?
ErrorSnapshotsdatastoresVM SettingsBrowseDataStore
0
Allen Scott
Asked:
Allen Scott
  • 34
  • 30
4 Solutions
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You have 3 snapshots associated with this virtual machine, which have filled up your datastore?

do you know what a snapshot is ? Do not meddle....

You can read my EE Article, to familiarize yourself with a snapshot.

HOW TO: VMware Snapshots :- Be Patient

and I can work with you to find the best possible solution.
1
 
Allen ScottAuthor Commented:
Andrew, yes please. I would like to find the best possible solution.
0
 
Allen ScottAuthor Commented:
So after reading, I think it may be safe to delete the 00001 & 2 files.  Correct?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So after reading, I think it may be safe to delete the 00001 & 2 files.  Correct?

No it is not, because the snapshots files are each chained... so if you delete a file, the chain will fail!

VM disk = parent + snapshot1 + snapshot2 + snapshot3

if you delete a snapshot the chain fails...

Okay, what virtual machines share this datastore1 ?
0
 
Allen ScottAuthor Commented:
I only have 1 VM.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I'm looking at the datastore, and it states it's 468GB

TSERVER is 200GB, so the other 200GB was it gone...

and what is RDM and localRDM1 >?

Does TSERVER have more than a single disk ?
0
 
Allen ScottAuthor Commented:
I dont know about RDM and LocalRDM.
Hard disk 1 - Datastore1 200GB
Hard disk 2 - Datastore1 160GB

Is there a chat program we can use?
0
 
Allen ScottAuthor Commented:
Both RDM and localrdm1 folders are empty
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
All communication at EE, is supposed to take place via the forum.

Okay, you have another datastore called WD_Black, to solve the snapshot issue, we must delete the snapshot correctly, and the method is to merge it correctly with the parent.

Is the VM currently off ?
0
 
Allen ScottAuthor Commented:
Yes, it is off.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, Right Click the VM, and select Snapshot > Take Snapshot

and then wait 2 minutes.
0
 
Allen ScottAuthor Commented:
done
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Before we continue do you backup VMs using any third party backup software ? e.g. Veeam etc
0
 
Allen ScottAuthor Commented:
I tried Altaro, but it failed.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
did this fail, when you were trying to back it up ?

where is Altaro installed on another VM on another host ?
0
 
Allen ScottAuthor Commented:
another host. It failed about 10 minutes into the backup.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, you need to check IF the Altaro VM also has attached to it, this VMs Parent Disk, and delete it from the Altaro VM, but do not delete from disk, just the inventory.
0
 
Allen ScottAuthor Commented:
Sorry, I think I might have misspoken.  Altaro is installed on another Physical Server (Not Virtualized).
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay now, Right Click the VM, and select Snapshot > Snapshot Manager > DELETE ALL

before doing this understand, that snapshot deletion can take a very long time, from seconds, minutes, hours, days or weeks, it depends on the size and number of snapshots and the underlying speed of the datastore.

BUT it does succeed, sometimes it can stick at 90, 95, 99, or 100%, do NOT be tempted to meddle, cancel, restart VM, host, vcenter, or any other service.

be calm, and go for a cup of java!
0
 
Allen ScottAuthor Commented:
Virtual Machine disks consolidation failed.
Remove all snapshots completed.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
can you check the VM folder, and also check disk settings ?

00001
00002
00003

should be gone...
0
 
Allen ScottAuthor Commented:
VM Settings now shows all my hard disks using a snapshot 00000#
and they still exist in the folder.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what is #

can I have a new screenshot or both disk settings and VM folder.
0
 
Allen ScottAuthor Commented:
1.PNG2.PNG3.PNGCapture5.PNG
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, it could not consolidate the disks, because there is a LOCK on the parent disk more than likely caused by the failure of Altaro.

We can start looking for the process which has the lock and kill the process, but this often provides a waste of time, as you only have a single VM on this Host.

Please restart the host which hopefully will clear the lock on the parent disk, and then we will be able to repeat the process above, and remove the snapshots correctly.
0
 
Allen ScottAuthor Commented:
Disks consolidation still failed
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
did you Take a Snapshot, and Delete ALL

has the snapshot increased to 0006 ?

do you have vCenter Server ?
0
 
Allen ScottAuthor Commented:
Yes, took a snapshot, delete all.
Now Disk 0006.

I am using vSphere Client.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you not have vCenter Server managing your hosts ?

do you have more than a one host ?
0
 
Allen ScottAuthor Commented:
No. We only have the one VM in our environment. It is running on an ESXi host. We use the vSphere client to access it.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So just a single ESXi host, and no vCenter Server ?
0
 
Allen ScottAuthor Commented:
correct.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
and you restarted this host ? because the lock is still present which is why we cannot consolidate the snapshots ?
0
 
Allen ScottAuthor Commented:
Yes, I restarted the host.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay....

Your datastore1 is quite a small datastore, to have a VM with two disks of

200GB
160GB

when, a SWAP file is also created on the datastore equal to the size of memory in the VM which is approx 50GB, so that gives a used space of

200+160+50 = 410GB out of 460, which gives us only 50GB working space, which is not a lot.

To see if the VM starts, we can reduce the size of memory, to 8GB, and start the VM. (this is just as a test, to check we don't get any strange errors! because the problem with running out of space like you did, could result in a corrupted snapshot chain)
0
 
Allen ScottAuthor Commented:
Just restarted it again.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
the HOST ?
0
 
Allen ScottAuthor Commented:
okay, changing the memory does allow the machine to turn on.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, that's good....

and it starts and powers up and you can login ?

what is the space free on the datastore ?
0
 
Allen ScottAuthor Commented:
Yes, I can login.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
free space on host datastore ?
0
 
Allen ScottAuthor Commented:
14.98GB
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, that's not much for us to play with....

Can you try and take a new snapshot and delete ALL
0
 
Allen ScottAuthor Commented:
While it is on?
0
 
Allen ScottAuthor Commented:
Do I want to check either of these?
Snapshot the virtual machine's memory
Quiesce guest file system
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Just this option - Quiesce guest file system
0
 
Allen ScottAuthor Commented:
While VM was on, took snapshot with that Quiesce guest only selected. Then delete ALL.  Still consolidation failed.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, there is a lock on the parent disk somewhere, and whilst this is present the consolidation will fail.

is the Physical Server that Altaro was installed on OFF ?

Okay, there is another option, we can use VMware Converter, installed on this VM, to create a V2V, e.g. a converted VM, almost a CLONE. This can be created on your other datastore WD_black.
0
 
Allen ScottAuthor Commented:
No. it is the same machine I am on to use the vSphere client.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Install VMware Converter on the VM

see my EE Article to obtain the software

HOW TO:  P2V, V2V for FREE - VMware vCenter Converter Standalone 6.1
0
 
Allen ScottAuthor Commented:
So,
Hard disk 1 & 3 is on datastore
Hard disk 2 is already on my WD_Black.

So how does that work? (Creating a V2V) to the WD_Black?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
but WD_Black has far more space than datastore1 which is not suitable too small for your VM.

VMware Converter, will create a new VM, with no snapshots, and after some minor modifications, after you have tested it will be able to delete the old version. (which exists as a backup!)

and all folders on datastore1 and disks.
0
 
Allen ScottAuthor Commented:
Can we do remote access support? Team VIewer?

I get the below
error.PNG
0
 
Allen ScottAuthor Commented:
Is the VM still functional during this V2V??
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, but do not use it..... otherwise your COPY will be out of date!

Hang on here a minute...

Your terminal server has three disks

Disk 1 - 200GB
Disk 2 - 1.2TB
Disk 3 - 160GB

You do not have enough free storage on WD_Black ?

Do you really need ALL that storage space for a terminal server ?
0
 
Allen ScottAuthor Commented:
That drive has 583GB used.
Can I make it smaller?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, currently WD_Black has 1.0TB FREE.

You can make ALL the disks smaller initially, so we can produce a COPY, and we can grow them later.

If you select EDIT, it allows you to customize the destination size of the disks.

I would change the size of them all, to ensure you can create a copy which will fit in 1.0TB, and allow size to grow and run the SWAP file there!
0
 
Allen ScottAuthor Commented:
alright. it is running now.
Thanks thus far.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
when the conversion has completed at 100%.

Make a recorded note of the Network Information e.g. TCP/IP Address, DNS, WINS etc - you will need this later to reconfigure the settings in the new VM

then power off the current VM, hopefully this is the last time we will use it.

Power on the new (copy) VM - use the TCP/IP information from above.

TEST the VM.... it will not have any snapshots, and will be "pure and clean"
0
 
Allen ScottAuthor Commented:
It is reporting 9 hours estimated remaining.
0
 
Allen ScottAuthor Commented:
Was the lock files because of the failed snapshot backup?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The lock is possibly caused by Altaro.

You would need to shutdown Altaro server, and then install vSphere Client on another computer, and check.

Find Locks on VMware VMs

see here

https://kb.vmware.com/s/article/10051

It can be quite time consuming...
0
 
Allen ScottAuthor Commented:
Thank you Andrew Very Much!  
The Cloning went well, however, it was not needed.  I went to where our 3rd party backup software was and shutdown all the services that was associated to it.  I then restarted the ESXi host and FINALLY saw that there were NO lock files.  So, with VM turned off still, took a snapshot, waited the 2 minutes, Delete ALL.  It did NOT error out and took about 45 minutes till it reached 100%.  I verified in the settings that Hard Disk 1 was on the correct vmdk (Did not have 000000#).  It also freed up just over 100GB on datastore1.  

Now, one more quick question.  Is there a place I can download or view all the logs the VM experiences and the actual host?

Thanks again!
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
/var/log/ on the host are where all the logs are stored for the host.

if you look at the VM Folder there is a log called vmware.log.

Glad you are all fixed, and thanks for coming back, don't forget to select a solution to close the question!
0
 
Member_2_4686089Commented:
Which log in the Var/log would I look through to see if there was power failure or if the host was restarted or if someone just held down the power button and turned off the host?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

  • 34
  • 30
Tackle projects and never again get stuck behind a technical roadblock.
Join Now