Vmware Consolidation needed, but fails with Input/Output Error.

Hi,

Can be very different problems.

Some inconsistent snapshot (also known as broken snapshot or invalid snapshot) between snapshots.

You need to edit each snapshot file and check the id from the previous one and the next one. Double check if all are correct, if not correct with the proper ID.

After that try to consolidate.

Note: This process needs to be done withe the VM power off

Other options is to vStorage motion the VM. Sometime this task correct the problem.

Another and lost option is to or either clone the disks, or use VMware Converter to convert the VM and have a fresh new copy with the Virtual Disks fixed.

Jail

Some background information, was the VM running and Powered On, did it fail with running out of storage?

has someone removed snapshots manually, by deleting from datastore ?

does the VM still Power On, or does it give an error ?

Do you have a full backup of the VM ?

Consolidation is rubbish, and very often fails. I can work with you to resolve the issue, but you must follow my instructions, and do not stray off the path.

You have 7 snapshots, these could take some time to merge.

what space do you have free on the datastore ?

Can you try the following:-

0. Stop all backup Jobs.
1. Power OFF The VM.
2. Take a new snapshot.
3. Wait 60 seconds.
4. DELETE ALL from Snapshot Manager.
5. WAIT BE Patient.
6. It could take seconds, minutes, hours, days, or weeks to merge.

this is based on speed of datastore, do not be tempted to cancel, mess, meddle, shutdown host, restart VM etc

it may appear to hang at 90, 95 or 99% this is normal.

Let me know the results.

Can you post me a screenshot, and then we will move onto the next check..... where we can verify the vmdk file.

Hi thanks for reply. For your question.

1. VM is running normally, i can shut it down and run again. For now :)

2. As i know no one removed any snapshot manually but it is possible.

3. No i dont have a full backup and cloning vm fails too :(

4. I have 1,3 TB free space on storage,

5. Ok i will try it now. Post results later.

Thank you.

Do you have backup running on that VM??
Sometimes backup do the same behavior like that on VMs.

Cloning VM should not work and also if vmotion to a different Datastore not work, first you could try to clone the virtual disks.

But first before anything, I would go to any snapshots one but one a check the id for each one if is correct and pointing to the right snapshots(check CID parental)

Here you can see how to check CID https://kb.vmware.com/kb/1007969

then after checked this try any other solution.

Jail

Completed succesfully.
But consolidation needed warning is still there.

Thanks Luciano i will try to check it ;)

Check if the VM is stll running on the snapshot...see my EE Article

HOW TO: VMware Snapshots :- Be Patient

Consolidation is DUMB and it appears if it detects a snapshot, it does not necessarily check if the VM is actually running on it!

they could just be orphaned files!

Check the VM disk! If it's not running on a snapshot, then there is A question?

I would also verify the snapshot (virtual disk) chain, you need to be at the console or remotely to do this..

also try this

Vmkfstools –t0 –v10 lastsnapshot-00000n.vmdk

Select allOpen in new window

this will check and verify the snapshot chain and virtual disk, and confirm, if it's correct or corrupt.

Also are you aware at present of any virtual machine data loss ?

what function does the VM perform ?

This is terminal server of our organization :/ Windows Server 2008 So yes i am aware of loosing it :)

I meant, are you missing any data at present ?

is it running on a snapshot > check virtual disk > send me a screenshot ?

Hi

Also the question that needs to ask is, who create those snapshots?? Was it created manually?? Or by any backup software?

Jail

Hi. Snapshots was created manually.

Hi Andrew no, not any data loss at the moment...
And for your question about virtual disks -

Hard Disk 1 -
[datastore1 (1)] SKBAS8x64/SKBAS8x64-000008.vmdk

Hard Disk 2 - [datastore1 (1)] SKBAS8x64/SKBAS8x64_1-000001.vmdk

I have deleted any snapshots in Vcenter client two day ago.

okay, so your VMs are running on snapshot disks.

it's weird that they are running and power up, because if the snapshot chain was invalid, you would get an error at power on.

okay, validate the snapshot chain, and verify the disk.

Vmkfstools –t0 –v10 SKBAS8x64_1-000001.vmdk

Vmkfstools –t0 –v10 SKBAS8x64-000008.vmdk

Select allOpen in new window

two commands there to validate both disks.

if they come back with a failure.....get and use "VMware Converter" and get a V2V quickly.

https://www.experts-exchange.com/articles/25539/HOW-TO-P2V-V2V-for-FREE-VMware-vCenter-Converter-Standalone-6-1.html

Install VMware Converter on the server, V2V, create a new VM, and dispose/delete of the old.

So you have a list of snapshots in Snapshot Manager showing presently ?

Are these commands safe? Should I first Clone disk just to be sure?
In snapshots manager there are no snapshots. Only "You are here" .

Pavol

After i run command:

Vmkfstools –t0 –v10 SKBAS8x64_1-000001.vmdk

Error - Extra arguments at the end of the command line.

Yes, these commands are safe to use, they will just verify the virtual machine disk.

okay, clone the disk at the command line.... using the following command, this will also complain if it cannot clone, it will also check the chain....once you have the new virtual machine disk, you can add to the machine, and then remove the other virtual machine disk, and snapshots.

vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

Select allOpen in new window

vmkfstools -i SKBAS8x64_1-000001.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk

vmkfstools -i SKBAS8x64-000008.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk

Select allOpen in new window

the above commands, will clone the disk, to a new disk, and merge all the snapshots at the same time, into a new disk.

let me know the results...

Can i do it when VM is turned on? I need to run it now some ppl need to work allready...

No, the VM must be off.

Also remember, when you start the clone, it could take minutes, hours, days or weeks to complete, when it's off.

this will make the VM unavailable.

If you have vCenter Server, you may be able to use the CLONE function, (Right Click the VM, CLONE), you must access the risk of users using the VM, versus total loss of the VM, if further corruption occurs.

Today this error pop up and stops the VM again... :( This Error allready shows two times before but i think with different vmdks. I power on VM and now running.


So what i will do first? :/ Check the wirtal disk with this command?

Thank you

Pavol

There is an issue with the Snapshot chain.

I repeat

do this quickly

https://www.experts-exchange.com/questions/28931056/Vmware-Consolidation-needed-but-fails-with-Input-Output-Error.html?anchorAnswerId=41494363#a41494363

and see if you get and error ?

Did you try and get a CLONE via vCenter Server, did it work or fail ?

You are now on borrowed time, of having a complete failure, and the VM will not restart and be trash!

or start that V2V NOW!

I tried cloning few days before in Vcenter and fails. Ok i am going to V2V now...
One question checking the virtual disks in comand line is useless now?

When i am trying to check disks ("vmkfstools –t0 –v10 SKBAS8x64-000008.vmdk") allways got this error :(

"Extra arguments at the end of the command line."

Can you help me pls?...

You've missed a post.....

please try these

vmkfstools -i SKBAS8x64_1-000001.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk

vmkfstools -i SKBAS8x64-000008.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk

Select allOpen in new window

different commands!!!!!

these will try and clone the virtual disk, but does depend on having a valid snapshot chain, these could also fail.... and then you will need to do a V2V ASAP!

Your VM virtual disk is corrupted.

I know, but i want to try check the disks as you mentioned before. So ok I am now trying to clone in Vcenter if fails will try to V2V...

let me know the results.

Cloning in Vcenter end with error :(

Will I try to V2V the VM now? Can be VM turned on?

Thanks

did you try at the command line with the commands I posted ?

otherwise you will have to get a V2V.... but it's possible that could fail when it tries to read the virtual disk, and get's to snapshot 8, which might need further work.

No i an not. So i will try now clone the disk with command line through ssh ok?

"vmkfstools -i SKBAS8x64-000008.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk"

I have a question. This "Temp folder name" is temp folder i will create on storage?

correct any folder will do, with any name you select.

Cant run command - error -

Destination disk format: Thin
Failed to clone object parameters

Its ok now. Clone is running now 4% done.

It will be interesting to see if it goes up to 100%, or fails, when it starts t read the last snapshot.

Failed :(

Clone: 30% done.Failed to clone disk: Bad file descriptor (589833).

You have no choice other than to try a V2V.

Hi, converting hangs on 1%, on operation clone the disk xxx.vmdk for a few hours. I try to disable SSL ( true to false) as you mentioned in other topic. But no success.

I try to convert directly from this corrupted VM also from my PC.

Should i try to run conversion from other VM on the same host? Or other converter version? I have 6.1.1.

Thanks

Install Converter on the VM to be converted, this is best practice and recommended.

Ok i will try tonight.

I am normally converting VM to VM in same host ok?
Not importing .vmx or...?

Can i do it when VM is on? Probably there will be perfomance issues...?

When covresion comes to 21 % error with corrupted redo log pop up :( :(
Tried 2 times .

There is no way to ged rid of that corrupted redo log file?

Please help...

I suspected as much, that it would pop up reading the corrupted area of the snaphot file.

options are you drop the last snapshot.the bad snapshot disk

try this command

vmkfstools -i SKBAS8x64-000007.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk

discarding the last snapshot, may cause corruption within the VM, but it looks like you have got no choice but to try this.

But i will run this command with SKBAS8x64-000008.vmdk or not?

I want to ask. And if i try to image disk with 3rd party app such as Acronis. And recreate virtual machine from image file?

Thanks

Please read my last post, NO YOU do not use snapshot it's corrupted, you ignore it, this is the issue with your VM.

vmkfstools -i SKBAS8x64-000007.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk

Select allOpen in new window

I would suspect if you try to read the entire disk, just like VMware Converter has tried, the same issue will occur, the error will pop up.

Your mileage my vary here, and you may end up with a working virtual disk, or it will be corrupted beyond repair, and you will need to restore the VM from a backup.

Problem is that i dont have a valid full backup of the server. I tried to backup with windows server backup but error with redo log pop up at 26 %. Just coincidence or do you think it is related to corrupted vmdk?

Thx

Now its bad. Redo log error pop up few seconds after VM boot. :( So i am going to try coomsnd with 00007 vmdk

Yes, the issue is, everytime you start the server, it's writing or not writing to the corrupted snapshot, number 8.

but, and this is a big but, with the snapshot chain incorrect, without snapshot 8, you may not have a valid NTFS partition, or it has corruption.

This is the danger of leaving a VM run on a snapshot disk for too long, especially with 8 snapshots, they get corrupted easily.

Part of every VMware Admin should be to check VMs daily for snapshots, and remove them, when they find them.

So this is my last chance?

vmkfstools -i SKBAS8x64-000007.vmdk /vmfs/volumes/<temp folder name>/SKBAS8x64_1.vmdk

No other solution? I am afraid of complete loosing of VM :(
Update of Esxi 5.5 (version 133182) is useless right?
Migrate to other host?
Create VM from orig vmdk?

I know it is probably not possible but i am just wondering :(
Pls for reply...

If you cannot use VMware Converter, Acronis, Windows Server Backup, to read the entire virtual machine disk, because there is a software fault at the virtual machine disk layer e.g. SKBAS8x64-000008.vmdk is corrupted...

it cannot be cloned, either via vCenter Server, or manually using vmkfstools, vmkfstools also reports an error in the snapshot chain.

SKBAS8x64-000008.vmdk, where the corruption occurs, needs to be discarded.

The question is what data is in SKBAS8x64-000008.vmdk, which maybe needed by the VM, to prevent corruption at the OS partition layer.

Keep you finger crossed, that this can be cloned excluding snapshot 000008, and you do not have any OS partition corruption.

Let me know the outcome of this operation above, e.g. clone using snapshot 0007.

Upgrading host, migrate to other hosts will fail, you'll not be able to add vmdk to another VM, and the issue is with snapshot 0008 corrupt.

Ok i am going on it... Wish me luck...

Hmm :(

Clone: 31% done.Failed to clone disk: Bad file descriptor (589833).
Same error as before only 1 % above.

Command - vmkfstools -i SKBAS8x64-000007.vmdk /vmfs/volumes/Datastore03/Clone/SKBAS8x64_1.vmdk


I wonder if there is not a storage problem?

Looks like the entire structure is at risk.

you could keep going, e.g. use the next snapshot

SKBAS8x64-000006.vmdk, and repeat, until successful, with 5,4,3,2,1

if you have time, it might be worth checking the snapshot chain.

see here

https://kb.vmware.com/kb/1007969

but this is a little odd, because I would have thought if the CIDs were mismatched, it would not Power On.

Odd is there is the same "Bad file descriptor (589833)." right?

I have time till morning :) 7 hours from now.

I am reading now this article https://kb.vmware.com/kb/1007969 but can you help me with it? I am little lost there :(

So what do you prefer to clone the disks 5,4,3,2,1 or to check CID?

So not to check CID? What do you think?

no don't bother, try and get a clone of the disk.

Ok i am now trying 6

same bad result :( so try 5?

yes, carry on discarding snapshots.

is it always at 30%

Yes ok. Do you think there is still way to retrieve a VM that will be running ok? :( :(

Give me hope please :)

We can only keep discounting and removing snapshots...

It failed with xxxxx3 .vmd too. I try to do this with 000002 but it drops my internet connection so right now i dont know if it is running now or it failed. Is there some way to see the results or if it is still running?

Or do it again?

Thanks

Friend give me an advice to:

Create New Virtual Machine With Same operating system and same Hardware specs.
Create another temp VM.

Mount on temp VM HDD from corrupted VM and HDD from New with same OS.
Delete all files on new HDD and clone it with "Robocopy".
What do you think about it?

Update: VM running in safe mode ok.

Anything is possible, but I think you may find, when you get to the area of the corrupted virtual disk, the same issue will occur, and the error will pop up.

check the output for a new file, but I would re-run again. if it's failing this quick, it will fail again quickly or continue.

Hi i try some things to map hdd to another VM. (orig was off)

Now i have this error...

Cannot open the disk '/vmfs/volumes/503d9cdf-b48c7a00-76e8-6c3be5beaa34/Zaloha/SKBAS8x64-000008.vmdk' or one of the snapshot disks it depends on.
The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child

looks like you powered the parent disk only, and not the snapshot, virtual machine disks are not really transportable, when they have snapshots.

how did it go with 0002 and 0001, be careful you are not just digging a bigger hole for your self!

at the very beginning, we did advise, do not stray off the path!

So any advice from you? :(

So cloning 000002 fails too...

I look at CID and 00001 have a bad parent cid.

SKBAS8x64 should be parent of 00001 right?

I need right now change the parent CID of 00001 right?

/vmfs/volumes/503d9cdf-b48c7a00-76e8-6c3be5beaa34/SKBAS8x64 # grep -i CID SKBAS8x64.vmdk
CID=0dbcc200
parentCID=ffffffff
/vmfs/volumes/503d9cdf-b48c7a00-76e8-6c3be5beaa34/SKBAS8x64 # grep -i CID SKBAS8x64-000001.vmdk
CID=d1aea23b
parentCID=5a94f981

did cloning fail before, you made these changes?

did you try 0001, ?

as for the current issue, with the mismatched CIDS, you now need to follow the VMware KB, I previously posted.

you need to ensure now, that ALL the CIDS match, all the way through the chain.

No i did not try 0001. Try it now? Or first change Parent CIDs?

okay, if you've not tried it, it's not going to work now, because there is a mismatch between parent and snapshot, e.g. the CIDS don't match!

Hi guys i was reading this article and few other about correcting the CIDs.
One thing is little confusing. I am going to edit only ".vmdk" files as a configuration right? Not "vmdk.delta"
Sorry if it is a stupid question i just want to be sure :) :)

Thanks

Hi

Sorry to not returning to this post, but since Andrew did follow up, you are in good hands

No only vmdk is to check.

Jail

Its ok thx

you will see there are two files, always with a vmdk, there are a pair, the descriptor file, which is a text file, which describes the geometry of the disk, and the actual data part.

so the small file, a few bytes, is the the descriptor file, which contains the number (CID) which needs to match.

Sok i repair missed CIDs and now is back in situation with corrupted redo log of 000008... :)

did you try the cloning 0001?

the next option would be to just discard ALL the snapshots, 1,2,3,4,5,6,7,8.

and then remove the virtual disk from server, and add back SKBAS8x64.vmdk.

However, your results may vary, and the OS partition may be corrupted missing data, files.

But I'm afraid the corruption occurred long ago.

Ok i will try it.
Just i case on server is very few valuable data. All data is on file servers etc...
I need to run OS "normally". This is our goal :)

And as i said it is strange that OS is running in safe mode OK. :/

Just to know, what are the age of those snapshots?

At least the first ones??

Jail

You may have no choice, but to start creating a new server.

Luciano age i dont exactly know age of snapshots. I may be months or less.

I am going to try 000002 once more and then 000001. Post results later...

00002 fail trying 00001 now

cloning 000001 done OK. What to do now please?

Okay, so remove the bad virtual disk from the VM.

and add the new disk you have created to the VM. (e.g. the cloned)

Hmm it looks like OS is corrupted. Only Black screen. I do chkdsk and now blue screen after windows logo. Any tips? :(

Consolidation warning is still there...

Yes, I suspected as much, without the full snapshot chain, the OS partition would be corrupt.

The Consolidation warning, is likely to be there, because it's stupid, and just pops up, when it detects a file with 00001 etc

I'm afraid, you will have to restore from backup, and if you have no backups, re-create your VM.

The virtual machine disk, was corrupted long ago, and snapshot corruption cannot be repaired.

I am trying to repair my Windows Server 2008 but i think that system files are too corrupted.
Any way i manage to run the corrupted VM now for a few hours so i can transfer TS Licences to another server and do what is necessary.

Anyway Andrew i want to thank you for your time and help. I learn so much,

Pavol Vrabec

Update:

This maybe help somebody to run server temporarily.
Yesterday i disabled some services that are not essential needed for OS and disable Antivirus.
Now server running for 12 hours. I am now preparing new server.

Pavol

I'm sorry the outcome was not what you expected, be careful with snapshots.

and monitor them in the future.

Yes you are right. Sorry for mistake. Thx