vmdk_00001 keeps growing not sure why

greetings,

vm_vmdk_0001 continues to grow. I have to keep adding space to the datastore. veeam ran and completed successfully. it usually creates a vmdk_00002 but then removes it. not sure if it also creates the vmdk_0001, just never noticed. any idea why 0001 keeps growing? is it associated with veeam backup or is it a normal file of the vm? it is about 28 GB and growing. I have my vm.vmdk showing as the size of the vm (540 GB).

thanks
LVL 7
king daddyAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

coolsport00Commented:
It needs to be removed. It's a snapshot and will continue to grow until removed. Rt-click on the VM -> Snapshot -> Snapshot Manager and select 'Delete All'. This will "commit" all the data in the snapshot to the snapshot's 'parent disk' (the orig VMDK it seems like). Once all the data is 'committed', then the delta snapshot file will be deleted.

~coolsport00
king daddyAuthor Commented:
OK. just checked snapshot manager and it shows a snapshot associated with veeam and to not remove. however, the veeam backup completed successfully so not sure why it wasn't removed.
coolsport00Commented:
Is your Veeam server a VM?
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

king daddyAuthor Commented:
veeam html report shows this:

Backing up object "[EXCH1aEMC] EXCH1a/EXCH1a.vmdk"
Hot add is not supported for this disk, failing over to network mode...
VSSControl: FinishSnapshot failed. Transaction logs will not be truncated.
RPC function call failed. Function name: [FinishSnapshot]. Target machine: [192.168.25.5].
RPC error:The RPC server is unavailable.
Code: 1722

Removing snapshot
RemoveSnapshot failed, snapshotRef "snapshot-4515", timeout "3600000"
A general system error occurred: Invalid fault
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually.
Possible causes for snapshot removal failure:
- Network connectivity issue, or vCenter Server is too busy to serve the request
- ESX host was unable to process snapshot removal request in a timely manner
- Snapshot was already removed by another application


so if it is stuck, how can I remove it?

thx
FostWareCommented:
Double check that Veeam is clearing it's snapshots.

VCenter should list "reconfigure guest", "remove snapshot", "reconfigure guest" events on that VM.
If not, I'd raise a query with Veeam support
coolsport00Commented:
If it is, AND you have "hot-add" configured, if you rt-click on the Veeam VM -> Edit Settings, you may see an extra 'hard disk' associated with your VM. First thing to do is click this hard disk and select Remove to remove it (***BUT DO NOT SELECT TO DELETE IT***). Once it is removed from the Veeam VM, then you are safe to delete the snapshot file in your original VM. I've done it plenty of times. And yes, Veeam is "supposed" to remove snapshots, but for some reason it doesn't. You may need to follow up with their engineering team to see why...
king daddyAuthor Commented:
yes, it is a VM
coolsport00Commented:
Ok, so hot-add isn't supported. No biggee. Go into Snapshot Mgr of the VM and select to Delete the Snap. It will seem "hung" at 95% for some time. Just be patient and let it finish. If the snapshot has been there for a while (a day or few), it may take a hour or 2 to finish...

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
king daddyAuthor Commented:
thanks all. headed to the office. will do what coolsport suggested asap.

thanks again
coolsport00Commented:
Keep us posted if that removal process works or not. If not, there are other things we can try...
IanThCommented:
is the vm hdd configured as thin provisioned?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It's a SNAPHOT, that's why it's growing....


A snapshot is NOT a backup of a VM; that is a gross misconception.  

A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes.  When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file.  There is an active link between the original VMDK file and the new delta file.  Anything that gets written to the VM actually gets written to the delta file.   The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM.  If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.  

I highly recommend reading these 2 articles on VMware Virtual Machine Snapshots:

Understanding Snapshots - http://kb.vmware.com/kb/1015180
Snaphot Best Practices - http://kb.vmware.com/kb/1025279

Also check out the following Snapshot Articles by Eric Siebert

Pt.1- http://is.gd/Lajg4p
Pt.2- http://is.gd/NdKQWC
Pt.3- http://is.gd/tp2vEK
markvorCommented:
The vmdk is from a not successful delete snapshot from veeam.
So here what you have todo:

1.Be sure that the Backup Job is completed (failed)
2.In the Settings of the Veeam-VM, remove (!!!not delete!!!) the the vDisks.
3.Eventually make a Dummy Snapshot on the Veeam VM
4.Open Snapshot Manager in the Veeam-VM and delete all Snapshots
5.Open the Settings of that VM which vDisks are removed from the Veeam-VM (under Point 2) and do a Dummy Snapshot on that VM and delete all Snapshots from the VM  right after the Dummy Snapshot is done.


Hope that helps

Best regards

Markus
king daddyAuthor Commented:
is it safe to delete the veeam snapshot? exchange is down as it is out of space again so need to delet ethis quick..

thx
coolsport00Commented:
Yes...it is...

You have a b/u of the VM (I assume), so you have that to go back to if somehow the 'commit/delete' process messes up your VM.
king daddyAuthor Commented:
I do have a backup from last night. it actually shows veeam successfully completed, but there was a warning. I pasted it above.

so do I do a delete or a delete all on the veeam snapshot
markvorCommented:
yes
coolsport00Commented:
Yes, delete all - what that does is "commit" (save) all data that has been added to the snap file since the snapshot was take back to the snap's parent disk. Once all that data is commited/saved to the parent disk, then the snap file will be deleted...
markvorCommented:
which version of vmware is installed
king daddyAuthor Commented:
just did it. hope I have enough room on the datastore to handle this, if the datastore is even affected. snapshot was about 37 GB, so it may take a while.

thanks everyone for the quick and helpful responses. will post back when complete.
markvorCommented:
be sure to wait for the complete statement also when the timeout error comes
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Be Patient, and try not to cancel any progress on the server!
king daddyAuthor Commented:
will do. however, now I have a vmdk_0002 file that is about 2 GB and growing. I only have about 7 GB left on the datastore and only about 5 GB left available on the SAN for this datastore. worried this will not be sufficient for the snapshot removal process. may have to ask another question about how to add space to this datastore from another set of disks on the SAN, if that is possible.

also, should my vmdk_0001 be shrinking yet or not until it fully commits?

thanks again
markvorCommented:
which version of vmware is installed ?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes you can extend datastores
markvorCommented:
It depends on your storage and vmware version
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
until the data is merged, the vmdk will not disappear
king daddyAuthor Commented:
markvor - ESX 4

hanccocka - thanks. I will post a link here for guidance on adding space from another set of disks on the SAN to this VMs datastore as I am now down to about 5 GB.
markvorCommented:
Ok hope the space is enough.

In case u can not extend your volume you can do the following.

Clone the virtual machine to a different datastore (you can select a different destination for each virtual disk in the clone wizard). All of the snapshots are committed to the clone virtual machine.

If you have vCenter Server 4.x, you can prevent this issue by performing the steps in Configuring VMware vCenter Server to send alarms when virtual machines are running from snapshots (1018029
markvorCommented:
Only for your information how Vmware 4.0 and earlier works with snapshots

When using Delete All in the Snapshot Manager, the snapshot furthest from the base disk is committed to its parent, causing that parent snapshot to grow. When that commit is complete, that snapshot is removed and the process starts over on the newly updated snapshot to its parent. This continues until every snapshot has been committed. This can lead to an aggressive use of additional disk space if the snapshots are large. Use care when exercising this option if there is not much space available on the datastore.
markvorCommented:
To extend the VMFS volume:

Select the host on which the virtual machine resides and click the Configuration tab.
Select the datastore on which the virtual machine resides and click Properties.

Note: If there is no available storage, a new LUN must be presented to every ESX host that can see the LUN.

In the dialog that appears, click Add Extent and follow the prompts in the Add Extend wizard to add an extent.
Perform a rescan on every ESX host that is being presented the new LUN so that the addition of the extent is detected.
After you have extended the VMFS volume, you can check the Retry option of the Redo log pop-up.
king daddyAuthor Commented:
markov - please copy and paste your response above about adding the extent to this question. it definitely deserves its own. thanks

http://www.experts-exchange.com/Software/VMWare/Q_27652388.html
king daddyAuthor Commented:
meant markvor
king daddyAuthor Commented:
so the vmdk_00001 that reached 37 GB has been removed. the vmdk_00002 reached 12 GB but stopped increasing, but now there is another vmdk_00001 file that shows a created time of 1:02 PM today and it is growing, rather quickly I might add. About 12 - 15 MB every few seconds (likely more around 20 - 30 seconds). I click refresh on the datastore 3 or 4 times and can see the increase.

Any idea what is going on with that? Am I in some kind of snapshot loop? just made that up, but it feels like a snapshot downward spiral culminating in the crashing of my exchange server!
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It looks like your VM is stuck writing to snapshots, rate of snapshot growth is dependant upon on how many changes in the VM.

file servers and email servers change a lot by the hour.

if Veeam is not currently backing up the VM.

Do these Snapshots appear in Snapshot manager?
king daddyAuthor Commented:
only the Veeam snapshot appears in the snapshot manager. would rebooting do anything for this?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I do not think a reboot will solve this.

The virtual disk of this VM is not connected hot added to Veeam Backup VM?
king daddyAuthor Commented:
no, it is not hot added. hot add always fails and switches to network mode.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, you need to Delete this Snapshot.
king daddyAuthor Commented:
that option is not available in snapshot manager, but that is what I did this morning around 9:30 AM. after clicking delete all on the veeam snapshot in snapshot manager, a vmdk_00002 snapshot was created.

the vmdk_00001 stopped growing and vmdk_00002 began to grow. vmdk_00001 was removed around 1 PM and vmdk_00002 stopped growing but another vmdk_00001 was created at the same time (around 1) and continues to grow and is now almost 8 GB.
king daddyAuthor Commented:
same thing just happened. vmdk_00002 was removed. the 8 GB vmdk_00001 snapshot I referenced in my previous post has now stopped growing and a new vmdk_00002 snapshot has been created. it's almost 2 GB already.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
if you create a snapshot manually , wait 60 seconds, delete all
king daddyAuthor Commented:
take snapshot option is not available (grayed out). pretty much all the options are grayed out except open console and a few others.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, you got a few options...

1. command line use of snapshot
2. vSphere vCenter CLONE VM option.
3. vSphere vCenter vConverter Standalone to CLONE.
4. VMware Support, that may suggest 1, 2 or 3.
king daddyAuthor Commented:
option 1 > I guess I could do a cmd line snapshot, but never have and kind of want to stay away from snapshots unless you suggest this as a first choice.

Option 2 above to Clone through vCenter is grayed out.

if I use vConverter stand alone, do I clone (I realize that is what you stated), or can I do a standard V2V? what would be the difference?

I'll skip option 4 for now.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
V2V or CLONE much the same, but different terms.

very odd that clone is grayed out?
king daddyAuthor Commented:
all power, all guest, all snapshot manager options, migrate, clone, template, FT, all grayed out.

when I clone/V2V, should I power down? I hate to but this warrants a shut down as I don't want any problems with these snapshots while powered on. since I can power down, are there any options you would suggest that may be available since the VM is powered down?

to note, it just did the same thing. vmdk_00001 was removed, vmdk_00002 stopped growing at about 4 GB and vmdk_00001 was created is now about 2.5 GB and growing much faster than the previous ones.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
as this is an exchange server it would be better to clone cold, or at least on, but ALL services STOPPED.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
what version of exchange?
king daddyAuthor Commented:
I think I'll just do it cold. it is exchange 2003 (unfortunately).

crazy question, is it possible that the clone/V2V will detect the free space from the 1221s in the stores and let me reduce the size of that drive, thus reclaiming that space (meaning I could reduce, clone, then use diskpart to add more space back for growth, thus reclaiming the free space in my stores)? I doubt it works that way but thought I would say it out loud so-to-speak, even at the risk of sounding like a complete idiot.

just happened again. vmdk_00001 that I stated was growing fast stopped at 3 GB when vmdk_00002 was removed and now a new vmdk_00002 has been created and is about 1 GB now. so weird.
markvorCommented:
take a look at te latest vmware.log of the virtual maschine
there you can see what the vm does now
i thin the machine is currently removing snapshots

as you see the files are getting smaller
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
you could......

Create new Exchange 2003 Server, correct service packs for OS and Exchange, and move the mailboxes.
markvorCommented:
to do a cold clone could be a very hard thing cause the vmdk organisation of your vm is in a mess of normal vmdk and snapshots.
take a look inside the vmx file of the vm and you will see which disk are involed.

the buttons are greyd out, cause of deleting snapshots

is the vm already running ?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
cold clone boot cdrom in VM, and it images out the data.....to new VM with zero snapshots, and new VMX, coldclone does not know VM has snapshots
king daddyAuthor Commented:
I thought clones / V2V removed snapshots as I have had to get rid of snapshots not properly removed by veeam.

funny you should mention a new server. I was going to either do that or add another drive to exchange, create new stores, and move mailboxes. however, we have been doing some mass deletions / archiving firm-wide, which is not helping this old exchange server. I was hoping to wait until next weekend to create the new stores / server, after most of the deleted items were cleared and after most people were down deleting / archiving. I really hate creating a new exchange server as this last one (when I went from p2v) was a pain.

any preference regarding new stores on new disks or a new server?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
VMware CLONE function, snapshot + parent disk = clone

V2V copies the data within the VM to a new VM. New VM is a new VM, vmx, with data.


Personally, I would prefer a CLONE rather than V2V, but CLONE may not work. So then you got to do a V2V this could take a while. Datastores would be down. No email for business.

I could get a new Exchange 2003 server in and built, before the V2V had finished.

If you wanted, I could try an get the Top Exchange Experts in on this Q. they may have other ideas, of maybe Disaster Recovery or Move Mailboxes.

Rate of change of moving mailboxes, will swell the snapshots,

how many mailboxes, how many users, size of store, do use OWA, Activesync?
king daddyAuthor Commented:
75 mailboxes + system, 75 users, about 500 gb. Owa.

I'll ask another question and post link here regarding new exchange server. I'd love exchange experts help if you can rally them. Thanks!
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
drop me the link here, in discussion with them.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would opt for new exchange server.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay Exchange Experts prompted that new Q. coming.

I would recommend new server, move mailboxes, reconfigure owa, then retire old server.

Ill leave then to Answer...
king daddyAuthor Commented:
Thanks a lot. Driving now. It'll be later tonight.

Thx again
king daddyAuthor Commented:
well, got home and surprise, surprise, all snaps removed, all options back for VM, and plenty of space in the datastore. I am hoping that I don't have to go through this tomorrow after Veeam runs tonight.

Coolsport00 - when you mentioned patience, I didn't realize I'd have to mimic that of a Buddhist monk!

markvor - you were right, I didn't check the logs but it was indeed still removing snaps and creating new ones along the way. maybe because so much changes on Exchange.

hanccocka - thanks for your help as well. I will post the link for the Exchange question later this evening. I am still going to do that as the stores will be a mess after all the deletions and archiving going on. it may be next week before I actually move the mailboxes, but I am going to get the server set up and ready to go.

thanks as well to all who replied and tried to help with this.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
it can take hours, days, for snapshots to clear........

task in progress, hence all options grayed out.

I would get to the bottom of snapshot issues with Veeam, otherwise it could be back at next backup!
king daddyAuthor Commented:
Yeah - that's what I'm afraid of. Either way it's time for a new server, so I appreciate you offer to help with that. Also, it may be due to those symmpi errors I posted about where you mentioned it's likely the ms iscsi initiator. I still have to check that out.
coolsport00Commented:
Sorry...been away for a bit.

Yep...can certainly take a while. I might also suggest to contact Veeam as to the cause of why not able to be able to remove the snap as it was supposed to.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
the problem with the snapshot method used by ALL backup software, is when the snapshot is started, and your VM starts writing out to the delta snapshot, the larger the snapshot, the longer it takes to committ, delete.

if you schedule your veeam backup jobs in hours or during very busy exchange times, your snapshot will be very large, and it could have issues deleting at closr of backup, then just gets worse!

So I would recommend setting Veeam Backup Jobs at the quietest times!
coolsport00Commented:
Well, since it was early this a.m. when it started (at least my time :) ), my assumption is that is started overnite? And, with that, little data changing? @hanccocka, I really love Veeam, but I have had this issue (snap not being removed properly) more times than I would've liked, and ALL my jobs (repl & b/u) are run overnite. It would happen once or twice within a week, then not happen for a month or 2. So, I would just either let Veeam try to rectify itself on subsequent b/u runs, or just remove the snap myself.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
All products have always had this issue, seen it every year, for many years...

hence why many of our clients and us, dont VMware Snap anymore, SAN Snapshot instead, big production systems.
markvorCommented:
eventually it is possible to get to a higher version of sphere because vmware changed the managing of removing snapshots in that version, also it is better to shutdown the vm before removing the snapshots, so the vm does not male any new snapshots.

only not veeam got this error, also symantec and many other products which were working with that technique are running in this errors.
king daddyAuthor Commented:
hanccocka - here is the link.

http://www.experts-exchange.com/Software/VMWare/Q_27654072.html

thanks again

I do start the exchange backup at 10:10 each night. it usually ends around 3 latest. the night it didn't remove the snap it ran until 5. it is still running now and veeam shows another 4 hours. that's about 14 hours. and it is running slow, 9 MB. looks like plenty of space now though.

again, thanks to everyone for your help.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
got it.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
the issue here, is your exchange server is running longer on a snapshot than on its parent disk.

longer the snap, larger the snap, more difficult to remove, veeam backup kicks in the next day creates another snap, so you are snap on snap, and we go round in circles....
coolsport00Commented:
Has this post morphed into another?...
king daddyAuthor Commented:
not really. I asked in another post about building a new exchange server as this one seems messed up. my question here was more about the snap not being removed, continuing to grow, and then creating new ones while removing others. please click the link and join my other posting regarding building a new exchange server if you like.

I will award points as soon as I get a minute to think about it. I hate awarding points as there are not enough for me to allocate for all the great responses.

thanks again everyone.
coolsport00Commented:
No worries..just seems this transferred from snaps to bldg a new server. I am traveling & not been able to follow as I normalky would so wanna make sure we're still discussing the snap issue . Seems like everyone's still involved. I won't be able to continue to  contribute till late tomorrow.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
yes, still, snaps......
king daddyAuthor Commented:
woohoo - it finished in 17 hours! sick. 9 MB. can't wait until tonight's back up in about 6 hours or so.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
any snapshots left?
king daddyAuthor Commented:
Nope
king daddyAuthor Commented:
so last nights took only 8 hours - so getting better.

thx again
king daddyAuthor Commented:
I believe the underlying resolution here, besides the obvious tech steps, is PATIENCE!

thanks everyone!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.