Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
SolvedPrivate

Consolidating VMWare Snapshots Not Attached To VMDK

Posted on 2014-07-08
32
Medium Priority
?
825 Views
Last Modified: 2014-07-14
Good morning,
We had a very interesting issue that I'll try to outline as much detail as possible:
Our Datastore continued to run out of space, which hosted a file server.  We expanded the Datastore a couple times and it kept doing it until we were out of space to allocate.  We didn't expand the guest (which has a 2TB volume), but it continued to run out of space.  What I noticed was a couple new hard drives (vmdks) in the folder.  They were attached to anything and we didn't run snapshots on the volume, so we were perplexed.  

Eventually, it was determined that our Unitrends backup system creates a snapshot to do its incremental backups, but it failed because there wasn't enough space on the volume and the snapshots just sat there.  Before we knew what the disks were we deleted one of them, unmounted the 2Tb VMDK and remounted it to get the server up and running.  Now there is a ton of 0KB data.  The one VMDK we understand is gone and we could restore some of the data, but there is now a 400Gb snapshot sitting there that I want to consolidate back into the 2Tb volume.

Is this possible?  The 2Tb volume doesn't seem to know that its missing any data or dependent on snapshots anymore.

I appreciate your help.
0
Comment
Question by:CorvalIT
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 18
  • 14
32 Comments
 
LVL 123
ID: 40183165
It could be possible which version of ESXi are you running because maximum virtual disk size is 2TB-512bytes including snapshot.
0
 
LVL 123
ID: 40183168
Unless ESXi 5.5 which Max is 62TB
0
 

Author Comment

by:CorvalIT
ID: 40183173
It is 5.5
0
Create the perfect environment for any meeting

You might have a modern environment with all sorts of high-tech equipment, but what makes it worthwhile is how you seamlessly bring together the presentation with audio, video and lighting. The ATEN Control System provides integrated control and system automation.

 
LVL 123
ID: 40183181
Okay you will need to edit vmdks and marry the CID mismatch to merge which could end up with corrupted parent.
0
 

Author Comment

by:CorvalIT
ID: 40183190
Ok, I'm not trying to corrupt my data... so real solution would be appreciated if there is one.
0
 
LVL 123
ID: 40183205
If you have a missing snapshot in the chain any merge could result in a corrupt disk.

Is the VM currently running on a snapshot ?
0
 

Author Comment

by:CorvalIT
ID: 40183216
No, which is why there's missing data.  No snapshots were ever created in VMWare

These were backup snapshots, so they don't exist in the snapshot manager (never did).  Unitrends creates a snapshot to backup the deltas and then deletes the snapshot.  In our case it didn't delete the snapshot, because it ran out of space on the datastore.  So all of our delta data is sitting in the snapshot of a failed backup.

I hope that makes sense.
0
 
LVL 123
ID: 40183222
Yes ALL Backup applications use VMware Snapshots to backup VMs and ALL backup apps leave VMs running on snapshots

Check your VMs daily set an Alarm in VCenter.

You deleted a link in the chain and the data is gone.

Force merge can be done but the result could be a corrupt disk.
0
 

Author Comment

by:CorvalIT
ID: 40183225
Is it possible to open the snapshot and get the data?
0
 
LVL 123
ID: 40183231
Which file did you delete?
0
 
LVL 123
ID: 40183233
No you cannot open the file.
0
 

Author Comment

by:CorvalIT
ID: 40183236
there were two vmdk's.  One was 146Gbs and the other 400Gbs.  I deleted the 146Gb volume.
0
 
LVL 123
ID: 40183239
Can u upload a screenshot of datastores?

I would post my EE article but on my mobile
0
 
LVL 123
ID: 40183240
Do you remember if it was 00001 ?
0
 
LVL 123
ID: 40183256
See my EE article about Snapshots because they are evil....


HOW TO: VMware Snapshots :- Be Patient
0
 

Author Comment

by:CorvalIT
ID: 40183258
2014-07-02-10-44-27.pdf

Attached as requested
0
 

Author Comment

by:CorvalIT
ID: 40183259
I actually read that yesterday, which helped me understand what the actual problem is.
0
 

Author Comment

by:CorvalIT
ID: 40183261
No, it was -000002.
0
 
LVL 123
ID: 40183266
Do you still have 0001?
0
 

Author Comment

by:CorvalIT
ID: 40183270
Yes, that is the 400Gb volume.
0
 
LVL 123
ID: 40183276
I would make a copy of the Files to work on to complete the merge.

As the result may or may not be corrupted?

Have you powered up the VM on the original parent since?
0
 
LVL 123
ID: 40183284
0
 

Author Comment

by:CorvalIT
ID: 40183285
The VM is running now on the original vdmk.  There are a ton of 0KB files, which I assume are on the snapshot.
0
 
LVL 123
ID: 40183297
You will be missing data whatever the outcome because you deleted 146GB of changes.

Which will be between the 1st and 2nd snapshot.

You may get some back in the 400GB if snapshot merges correctly without disk corruption.
0
 

Author Comment

by:CorvalIT
ID: 40183313
Ok, I'll try the article you mentioned.  

Is there a way to copy the 2TB vmdk to another datastore?  It would be quicker than a backup I would think.
0
 
LVL 123
ID: 40183412
You can either use cut/paste at the Datastore Browser or cp (at console/ssh remnotely)

Just for background understanding....

A virtual machine o parent disk looks like:-

Working.VMDK =  (parent vmdk + child snapshot-00001.vmdk +  child snapshot-00002.vmdk)

Changes after the first snapshot are created are written to:-

snapshot-00001.vmdk

Changes after the second snapshot is created are stored in

snapshot-00002.vmdk

The contents of the snapshots are block changes and deltas, when used with/merged with the parent complete the Working VMDK.

If a file is lost or corrupted, your working VMDK will have missing data.

It's possible to merge the 400GB back into the parent, BUT the results may not be correct.
0
 
LVL 123
ID: 40186437
You can use the vSphere Client, or use cp /vmfs/volumes/datastore name/vm folder name/filename /
0
 

Author Comment

by:CorvalIT
ID: 40187995
Ok, so I made a copy and changed the CID in the 400Gb volume to match the 2Tb volumes parent CID.  I removed it from inventory and readded it, and it powered up.  It doesn't appear as if anything changed though.  Can I consolidate it into a single volume now?
0
 
LVL 123

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 40188032
You can now try and select DELETE ALL, this will merge the contents with the Parent VMDK.

Also make sure, that when you Powered it Up, it should be "running" on this snapshot disk.

e.g. if you look at my EE Article, it shows how to check this.
0
 

Author Comment

by:CorvalIT
ID: 40195333
Ok, so here's what I did and was as successful as I could be given my ignorance.

1. I cloned the server to a new volume.
2. I went through and changed the CIDs and parentname per this article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007969
3. I removed the clone from inventory and readded it, verifying it was pointed to the snapshot disk.
4. I powered up the VM and saw the data was there with some corrupted data.  I restarted and ran CheckDisk, which cleared the bad data out.
5. I copied the data from the clone server (renamed, re-ip'd, etc) to the existing file server and everyone's data was back.

Thanks a ton Andrew!
0
 

Author Closing Comment

by:CorvalIT
ID: 40195338
Andrew nailed this out of the park and frankly I wouldn't have been able to do this without his help.  He saved my butt!
0
 
LVL 123
ID: 40195695
Many Thanks for the Kind Words!
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
This article outlines why you need to choose a backup solution that protects your entire environment – including your VMware ESXi and Microsoft Hyper-V virtualization hosts – not just your virtual machines.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question