VM fails to start due to lack of space

Hi,
 
 I have a VMWare server and I had two virtual machines (SBS2011 with 700GB and windows 7 with 60GB on a datastore1 with 1.08TB Volume (2 x 600GB HD).
 Yesterday morning and this morning, SBS2011 VM was shut down and it was complaining about "lack of space on the volume" and it failed to start this morning. So I have to move WIndows 7 VM to datastore2 to free up about 60GB of space in order to start SBS2011 VM. Then I opened the datastore1 and browsed the folder for SBS2011 VM and discovered there are three VMDK files: "D***S1.vmdk"(730GB) , D***S100001.vmdk (177GB) and D***S100001.vmdk (155GB) instead of one vmdk file.
 I suspect that these addtional vmdk files have created a shortage in space.
(1)  Why are these created and can I delete them?  
(2)  When looking at VM Edit Setting/Hard disk, it shows the main vmdk file is D***S1-000003.vmdk instead of D***S1.vmdk. Shouldn't D***S1.vmdk with 730GB be the main vmdk file for this VM?
VMWare DatastoresSBS2011 VM HD provisionSBS2011 VM Folder in datastore browser(3) When I look at datastore1 space status, it shows 300GB Free and that is misleading because all these additional vmdk files created within VM folder without me knowing it and I thought that I had 300GB of free space all along.
LVL 1
sgleeAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The first thing I can see is you have snapshots, two in fact which you need to deal with.

You should be checking your VMs daily, to ensure you have no snapshots.

see my EE Article

HOW TO: VMware Snapshots :- Be Patient

I can work with you to help you with the snapshot issue. DO NOT just DELETE the snapshot files!

These snapshot files, are left overs' from a backup application, unless you have created them.

1. You cannot just delete them, they needed and are called Delta files, Children, or Snapshots!

2.Correct the vmdk should be the parent disk, but not in snapshot mode, performance would have been terrible in snapshot mode, and you have two snapshots, so your VM has been writing  to snapshots for a while.

3. Have you used or created a snapshot, e.g. Take Snapshot, or do you use a backup application, that may leave a VM running on a snapshot disk, you need to check daily after every backup and check!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sgleeAuthor Commented:
Yes I have use "NAKIVO" backup program that runs once a week. But I have never created a snapshop manually. I suspect NAKIVO backup program perhaps did .

I will read article in your link above and get back to you soon. However my immediate concern is that SBS2011 VM might shut down again by itself, I am afraid, if no action is taken.

I discovered that It was shutdown (or disabled or locked up, however we want to call it) yesterday morning and it was shut down last night again. This morning I had to move the whole folder that belongs to Win71 VM (60GB) to another datastore to get SBS2011 VM up and running.
How much time do I have until another possible lockup?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, the back application would have left the VM, running on a snapshot, ALL BACKUP Applications do it occasionally, you need to check your VMs daily to check they are not running on a snapshot.

Okay, the method to solve this is as follows, do not follow any other methods...

1. Power OFF the VM.

2. Select the VM, Right Click the VM, Select Snapshot > Take Snapshot > Enter test name etc, do not tick any other options.

3. Wait at least 60 seconds.

4. Select the VM, Right Click the VM, Select Snapshot  > Select Snapshot Manager > DELETE ALL.

The function delete ALL merges all the child disks (changes since the snapshot was called), snapshot files, into the parent disk.

5. This can take many seconds, minutes, hours, or DAYS to complete.

6. IT WILL COMPLETE. BE PATIENT.

7. Go and Grab a Cold Beer, or Coffee, do not watch it...

8. Do not meddle with the VM, do not cancel, do not restart the VM, do not restart the host, do not do anything other than walk away, be calm...be patient....

9. If you mess, cancel, you could possibkly corrupt your VM, beyond repair.

10. It's possible it will appear to stick, at a random percentage and do nothing, e.g,. 16%, 95%, 90%, 99%.

12. Be Patient...and wait...
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

VACANCY007Commented:
Hello,

it ses that you've old Snapshots on this vmdk.
Take a look to the snapshots of this vm.
If you've enought space it shod be possible to merge this Files.

If you've licences in your ESX you could also try to do a storgage migration.

http://sostechblog.com/2011/02/01/vmware-repairing-orphaned-esx-snapshots/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1035550
0
sgleeAuthor Commented:
@Andrew
Step 2 is creating a new snapshot? If that is the case, would it reqiure more HD space? As you know, I am running out of space already due to these snapshots.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It will not create more space, when the VM is OFF.

This is why we turn off the VM, and leave it for 60 seconds.
0
sgleeAuthor Commented:
So, if I summarize the process,  we create a new snapshot while VM is turned off and then we delete them all, right?
I like to understand the reason why we have to create a new snapshop before deleting all existing snapshots?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So, if I summarize the process,  we create a new snapshot while VM is turned off and then we delete them all, right?

Correct, and select DELETE ALL from the GUI.

At present you will probably find, if you look in the snapshot manager, there are no snapshots.

By manually creating a snapshot, we are ensuring the snapshot chain is correct, which will allow us to remove, them from the GUI. So you will end up with a new small snapshot which is chained to the 2 snapshot, which is chained to 1st snapshot, which is chained to the parent disks.

DO NOT I REPEAT DO NOT DELETE THE FILES MANUALLY FROM THE DATASTORE BROWSER FROM THE DATASTORE.

THE PROCEDURE MUST BE TO REMOVE THEM (DELETE ALL FROM THE GUI).
0
sgleeAuthor Commented:
Fyi, I posted existing snapshot and there is only one. I was expecting more than one based on multiple vmdk files.
Existing-Snapshop.png
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
there are two on the disk, this is the problem, it's missing in the Snapshot Manager.....

look at the datastore, look for the files that end...

-00001.vmdk, and -00003.vmdk, these are both snapshot files!

it's often they do not appear in the GUI, because they were not created via the GUI, but via API calls to the OS.

Please carry on with the instructions.

1. Take a new Snapshot.....

2. then delete ALL
0
sgleeAuthor Commented:
Roger that. I will proceed and result the post.
0
sgleeAuthor Commented:
one final check:
I was on snapshot manager and ready to click [Delete All] button.
But then I saw "Test_July04_2015" that I created manually according to Step2 and it is highlighted.
Shoud I select/highlight the VM("D***1") name at the top before clicking [Delete All] button or it does not matter?
Currently  "Test_July04_2015" is selected as seen on the screenshot.
Temporary-Snapshop.png
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Shoud I select/highlight the VM("D***1")

select the top of the Stack.., and click Delete ALL.
0
sgleeAuthor Commented:
Snapshot ManagerShould I select "D***1" at the top of the snapshot GUI?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Correct
0
sgleeAuthor Commented:
Status: 17% done after 50 minutes.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, that's good....

at this rate of change.....

35% - 100 mins
52% - 152 mins
100% - approx 300 mins (maybe!)
etc
0
sgleeAuthor Commented:
After it shows 100% completion, what else do I need to check before I start the VM?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Check if the snapshots exist, in the datastore in the VM Folder.

and also check the virtual disk properties, and it should show the parent disk correctly, e.g. no -0000X.vmdk (where X is a number)

if all removed, Power ON.

Check daily after backups.
0
sgleeAuthor Commented:
Ok.
Is there a way to automatically to detect these snapshots other than manually checking VM folders daily?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You can set an alarm in vCenter Server, if you have vCenter Server, you can create and run scrips, or you can use commercial software such as

http://www.settlersoman.com/tools-for-monitoring-and-deleting-vmware-snapshots/

or the recent Snapwatcher App.

http://try.opvizor.com/snapwatcher/
0
sgleeAuthor Commented:
When I checked back about 30 minutes ago, it already showed "completed".
I checked the VM folder and all those extra vmdk files are gone.
Now it shows the correct vmdk disk name.
After rebooting SBS2011, I noticed a significant speed increase.

@Andrew
I appreciate your help.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Glad to be of assistance, keep an eye on those Snapshots!
0
sgleeAuthor Commented:
I will and thank you again!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.