Consolidating VMWare Snapshots Not Attached To VMDK

Good morning,
We had a very interesting issue that I'll try to outline as much detail as possible:
Our Datastore continued to run out of space, which hosted a file server.  We expanded the Datastore a couple times and it kept doing it until we were out of space to allocate.  We didn't expand the guest (which has a 2TB volume), but it continued to run out of space.  What I noticed was a couple new hard drives (vmdks) in the folder.  They were attached to anything and we didn't run snapshots on the volume, so we were perplexed.  

Eventually, it was determined that our Unitrends backup system creates a snapshot to do its incremental backups, but it failed because there wasn't enough space on the volume and the snapshots just sat there.  Before we knew what the disks were we deleted one of them, unmounted the 2Tb VMDK and remounted it to get the server up and running.  Now there is a ton of 0KB data.  The one VMDK we understand is gone and we could restore some of the data, but there is now a 400Gb snapshot sitting there that I want to consolidate back into the 2Tb volume.

Is this possible?  The 2Tb volume doesn't seem to know that its missing any data or dependent on snapshots anymore.

I appreciate your help.
CorvalITAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
It could be possible which version of ESXi are you running because maximum virtual disk size is 2TB-512bytes including snapshot.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Unless ESXi 5.5 which Max is 62TB
CorvalITAuthor Commented:
It is 5.5
Ensure Business Longevity with As-A-Service

Using the as-a-service approach for your business model allows you to grow your revenue stream with new practice areas, without forcing you to part ways with existing clients just because they don’t fit the mold of your new service offerings.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Okay you will need to edit vmdks and marry the CID mismatch to merge which could end up with corrupted parent.
CorvalITAuthor Commented:
Ok, I'm not trying to corrupt my data... so real solution would be appreciated if there is one.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
If you have a missing snapshot in the chain any merge could result in a corrupt disk.

Is the VM currently running on a snapshot ?
CorvalITAuthor Commented:
No, which is why there's missing data.  No snapshots were ever created in VMWare

These were backup snapshots, so they don't exist in the snapshot manager (never did).  Unitrends creates a snapshot to backup the deltas and then deletes the snapshot.  In our case it didn't delete the snapshot, because it ran out of space on the datastore.  So all of our delta data is sitting in the snapshot of a failed backup.

I hope that makes sense.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Yes ALL Backup applications use VMware Snapshots to backup VMs and ALL backup apps leave VMs running on snapshots

Check your VMs daily set an Alarm in VCenter.

You deleted a link in the chain and the data is gone.

Force merge can be done but the result could be a corrupt disk.
CorvalITAuthor Commented:
Is it possible to open the snapshot and get the data?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Which file did you delete?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
No you cannot open the file.
CorvalITAuthor Commented:
there were two vmdk's.  One was 146Gbs and the other 400Gbs.  I deleted the 146Gb volume.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Can u upload a screenshot of datastores?

I would post my EE article but on my mobile
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Do you remember if it was 00001 ?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
See my EE article about Snapshots because they are evil....


HOW TO: VMware Snapshots :- Be Patient
CorvalITAuthor Commented:
2014-07-02-10-44-27.pdf

Attached as requested
CorvalITAuthor Commented:
I actually read that yesterday, which helped me understand what the actual problem is.
CorvalITAuthor Commented:
No, it was -000002.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Do you still have 0001?
CorvalITAuthor Commented:
Yes, that is the 400Gb volume.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
I would make a copy of the Files to work on to complete the merge.

As the result may or may not be corrupted?

Have you powered up the VM on the original parent since?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
CorvalITAuthor Commented:
The VM is running now on the original vdmk.  There are a ton of 0KB files, which I assume are on the snapshot.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
You will be missing data whatever the outcome because you deleted 146GB of changes.

Which will be between the 1st and 2nd snapshot.

You may get some back in the 400GB if snapshot merges correctly without disk corruption.
CorvalITAuthor Commented:
Ok, I'll try the article you mentioned.  

Is there a way to copy the 2TB vmdk to another datastore?  It would be quicker than a backup I would think.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
You can either use cut/paste at the Datastore Browser or cp (at console/ssh remnotely)

Just for background understanding....

A virtual machine o parent disk looks like:-

Working.VMDK =  (parent vmdk + child snapshot-00001.vmdk +  child snapshot-00002.vmdk)

Changes after the first snapshot are created are written to:-

snapshot-00001.vmdk

Changes after the second snapshot is created are stored in

snapshot-00002.vmdk

The contents of the snapshots are block changes and deltas, when used with/merged with the parent complete the Working VMDK.

If a file is lost or corrupted, your working VMDK will have missing data.

It's possible to merge the 400GB back into the parent, BUT the results may not be correct.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
You can use the vSphere Client, or use cp /vmfs/volumes/datastore name/vm folder name/filename /
CorvalITAuthor Commented:
Ok, so I made a copy and changed the CID in the 400Gb volume to match the 2Tb volumes parent CID.  I removed it from inventory and readded it, and it powered up.  It doesn't appear as if anything changed though.  Can I consolidate it into a single volume now?
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
You can now try and select DELETE ALL, this will merge the contents with the Parent VMDK.

Also make sure, that when you Powered it Up, it should be "running" on this snapshot disk.

e.g. if you look at my EE Article, it shows how to check this.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
CorvalITAuthor Commented:
Ok, so here's what I did and was as successful as I could be given my ignorance.

1. I cloned the server to a new volume.
2. I went through and changed the CIDs and parentname per this article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007969
3. I removed the clone from inventory and readded it, verifying it was pointed to the snapshot disk.
4. I powered up the VM and saw the data was there with some corrupted data.  I restarted and ran CheckDisk, which cleared the bad data out.
5. I copied the data from the clone server (renamed, re-ip'd, etc) to the existing file server and everyone's data was back.

Thanks a ton Andrew!
CorvalITAuthor Commented:
Andrew nailed this out of the park and frankly I wouldn't have been able to do this without his help.  He saved my butt!
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization ConsultantCommented:
Many Thanks for the Kind Words!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.