Can not virtual machine consolidating

Hello all,
Last night Veeam Backup Job has given an error for a virtual machine. When I looked at this virtual machine in the morning, I saw that it was closed because there was no space on the hard disk (retry / ignore screen). however, there was enough free disk space. At the same time, Veeam could not delete the temporary snapshot that it created for this virtual machine.
for this reason the virtual machine needed disk consolidation.
I increased the disk size through the storage and I could open the virtual machine.
then I started the consolidation for the virtual machine with vSphere. but every time I got this error. "an error occurred while consolidating disks: 5 (input / output error)

I shut down the virtual machine and tried again.
I manually deleted the temporary snapshot, my attempt failed.
I tried to manually take the snopshot and it failed.
I manually deleted the SnapShot that I took manually and tried again.

how can I get rid of this situation? I really need help. because in this case the virtual machine is over-consuming hard disk. Today I made a 100GB disk increase in total, but it quickly consumed this space too.
Thank you in advance for your support.
Ali
ESXi 5.5
Veeam 7.0
Virtual Machine windows server 2008 x64
Ali VURALSystem ManagerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
did you check the virtual machine disk is still not attached to Veeam VM?
Ali VURALSystem ManagerAuthor Commented:
I find the virtual machine under "files" and when I look inside on Veeam, I see the .vmdk files for both disks connected.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If you edit the Virtual Machine for Veeam, do you see your virtual machine parent virtual disk, if you do select Remove from Inventory (do not delete from disk).

then proceed, to take a new snapshot, wait 120 seconds, and then select delete all in snapshot manager
Ali VURALSystem ManagerAuthor Commented:
Hi Andrew, thanks for support but I dont understand your first sentence.
Do you write the steps step by step and from which screen will I make it? I do not want to do anything wrong.
Will not I do the "add to inventory" process?
Do I need to take any action on Veeam Backup?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, please screenshot Veeam VM settings, e.g. Right Click VM, Edit Settings - Screenshot and upload here
Ali VURALSystem ManagerAuthor Commented:
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Ok SQL Server has snapshots

Can you do the same for your Veeam VM
Ali VURALSystem ManagerAuthor Commented:
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, if you

Is the VM powered on ?

1. Take a New Snapshot
2. Wait 120 seconds
3. In snapshot manager, select delete ALL
Ali VURALSystem ManagerAuthor Commented:
Yes, VM is the Power on.
I will do these steps then I wil inform you.
Thanks
Ali VURALSystem ManagerAuthor Commented:
hi, I comleted these steps.
R@f@r P@NC3RVirtualization SpecialistCommented:
hi Ali

If you can not consolidate and do not allow you to delete the snapshot manually.

It is possible that the snapshot is writing on the base disk, if the size of the snapshot continues to grow, if not, I recommend creating a folder in the datastore with a name of your preference and try to move that snapshot to the created folder.

Regards..
Ali VURALSystem ManagerAuthor Commented:
I did the consolidation, but I got the same error. the problem still continues
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
please can you screenshot your VM Folder
Ali VURALSystem ManagerAuthor Commented:
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay do you have vCenter Server ?

and what is the error message if any, when your delete ALL the snapshot ?

it is possible that you have a lock on the parent disk.....SQL.vmdk - which is causing the issue.
Ali VURALSystem ManagerAuthor Commented:
no, I dont have vCenter.
When I look at the VM files, I see a lock file as usual. can I get rid of this file solved my problem? I remember an article about it.
lock_file.JPG
R@f@r P@NC3RVirtualization SpecialistCommented:
Hello

Please share an error printer.

Additionally, connect to the ESXi where you have the VM, go to the datastore where the VM is hosted, create a folder that says LCK and move the file to the folder you just created.

do not create the folder inside the VM folder, create it outside and move the .lck file to that folder and try to consolidate.
R@f@r P@NC3RVirtualization SpecialistCommented:
o Perform these tests:

1- Create a snapshot with the option Quiesce guest file system.
2- Eliminates the snapshot.
3- Turn off the VM.
4- Edit the configuration of the VM - Go to advanced options.
5- Modify the value of the DiskEnabledUUID = False parameter. (If the value is not, add it).
6- Modify the parameter .ctkenabled = False. (If the value is not, add it).
7- Save the changes ..
8- Turn on the VM.
9- Migrate the vm to another host.
10- Turn off the VM.
11- Remove the VM from the inventory.
12- They try to consolidate.

Please hazem know the results.

Greetings.
Ali VURALSystem ManagerAuthor Commented:
Hello R@f@r,
I attached error message this post.cons_error.JPGThe VM running at now. What causes the interrupt on the VM, the process you recommended
Thanks
R@f@r P@NC3RVirtualization SpecialistCommented:
First, validate if in the veeam backup machine, go to the properties of the VM and verify if it has the SQL VM disk attached, validate that first, if you want to share the print...
Ali VURALSystem ManagerAuthor Commented:
I checked. The SQL disk is not inserted. 2 discs inserted and display images.

edit_settings_harddisk1.JPGedit_settings_harddisk2.JPG
R@f@r P@NC3RVirtualization SpecialistCommented:
ok, is your veeam backup server virtual? is vmware vm?

if so, I require that you go to the properties of the vm, and you share a printer of the vm disks of veeam backup

Greetings.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, please follow my instructions....ignore any previous instructions, and we can work to get this fixed.

The issue is likely you have snapshot corruption, and the chain cannot be merged. Please do not use CONSOLIDATE options.

Did you try


0. Power off VM
1. Take a new snapshot.
2. Wait 120 seconds
3. then select DELETE ALL, do you get the same error message ?

I'm not seeing any new snapshots created ?

because we will need to use a different technique to get this fixed.
Ali VURALSystem ManagerAuthor Commented:
yes, my veeam backup server is VM.
I attached screen shots of Veeam Backup Disk

edit_settings_harddisk1.JPGedit_settings_options.JPG
R@f@r P@NC3RVirtualization SpecialistCommented:
ok, your backup machine has not connected the disk of your SQL VM, we can already discard this part, I invite you to perform the tests mentioned above. and let me know the results.

1- Create a snapshot with the option Quiesce guest file system.
2- Eliminates the snapshot.
3- Turn off the VM.
4- Edit the configuration of the VM - Go to advanced options.
5- Modify the value of the DiskEnabledUUID = False parameter. (If the value is not, add it).
6- Modify the parameter .ctkenabled = False. (If the value is not, add it).
7- Save the changes ..
8- Turn on the VM.
9- Migrate the vm to another host.
10- Turn off the VM.
11- Remove the VM from the inventory.
12- They try to consolidate.
Ali VURALSystem ManagerAuthor Commented:
Andrew,
I cant power off it 1 hour ago. But I already tryied this method yesterday. But I got error mesages again in consolidation
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
just a quick question, this says vCenter ?

do you have a vCenter Server which manages the hosts ?

Okay, well snapshots could be corrupted and require a lot more work, which need the VM Powered Off and work at the console, or remotely via SSH?

Are you proficient at the console or remotely by SSH ?

do you want to continue with this fix ? Please note as your Virtual Machine si writing to a snapshot it will continue writing and expanding until all the space on the host datastore is full, and the VM will then fail and stop.
Ali VURALSystem ManagerAuthor Commented:
- I do not have a vCenter server.
- I'm not an expert on remote administration with SSH. (beginner level)
- I am aware of your concern. for this reason I want to find a solution as soon as possible
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, there are various technique we can try to recover your server.

1. We power off the server, and try some commands via SSH.

in the event this fails, we try a different procedure in the OS.

2. We create a V2V using VMware Converter to create a new VM.

I can hold your hand and guide you in the above step by step, but this is going to be a long process.

Otherwise I would suggest a Gig or Live Project on EE, to remote in and work and fix this for you.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ali VURALSystem ManagerAuthor Commented:
Hi Andrew,
in fact, I made your second suggestion to be an alternative solution. that is, I connected to the other ESX host by using vmware converter to convert the problematic SQL server to convert (V2V). I started this server by disabling the network board and saw that it worked.
I also see that the "need cosolidation" field is "NO".
if you think that this is a healthy method, I can use this new server to live and use it. It cost me only to restore only SQL databases.
I can also get rid of the problematic server after a while.
is it appropriate?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, that is a method you can use, make sure you check this new server you have created before you delete/archive/turn off the old.

new server will be snapshot free, I would suggest keeping for a few days (powered off) just in case you need something, or backup and archive!
Ali VURALSystem ManagerAuthor Commented:
Hello Andrew,
I solved the problem like this:
I created a copy of the problematic virtual machine with V2V and attached it to the other ESX 'host'.
I have restored the databases.
Currently the server is running, the users are connected and there is no problem.
Thank you for all your support.
Ali VURALSystem ManagerAuthor Commented:
Hello R@f@r,
I solved the problem as I explained above. I thought this was the fastest method. Thank you for your support.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Virtualization

From novice to tech pro — start learning today.