Problem with VM backups using Veeam

Hello all, I am having some problems backing up a few of my VMs (vsphere 5.0) with Veeam. I have tried looking at the logs and they are completely cryptic and I have no idea what they mean. I tried some google research and came up with nothing.

A few of my servers are coming up with this message.

Backing up object "[Admin-DS-11] Admin-Ajax-64Bit/QA-Dev Ajax-64Bit.vmdk"
BackupDiskFull failed
Client error: Failed to open VDDK disk [[Admin-DS-11] Admin-Ajax-64Bit/QA-Dev Ajax-64Bit-000002.vmdk] ( is read-only mode - [true] )
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [admin-vmh-11];Port: 902;Login: [root];VMX Spec: [moref=193];Snapshot mor: [193-snapshot-208];Transports: [nbd]] failed because of the following errors:

Server error: End of file


Now.. the kicker is my VBK files are being created, so it looks like it's being backed up, but I have not done a restore to make sure everything is working correctly. One of them is a critical server, so I am a little worried to do so with this error that's plaguing the backups for about 5 of my VMs. All the rest of my VMs (about 30 of them) are having no issues using the same job.

I have also created a brand new job with no luck.  Any help is appreciated please!!
kjs00333Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
As you have a snapshot on that VM, it might be worth getting rid of all the snapshots.

Can you confirm the snapshots exist, and size?

screenshot will do.

Please consult my EE Article

HOW TO: VMware Snapshots :- Be Patient

and Ask advice before Delting a Snapshot!
0
kjs00333Author Commented:
Nope, no snapshots at all. I made sure of it.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
how did you check?

this file

QA-Dev Ajax-64Bit-000002.vmdk

is a snapshot, the number 2 indicates there are two!

the destination location is not out of storage space?

the datastore where the VMs are located are not out of storage space?

what is the size of disk?

what is the max size of disk for the datastore? (blocksize)
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

kjs00333Author Commented:
I was checking in Vsphere on the VM itself under the snapshot manager. It's not showing in there at all.. is there another place I can check. The VMs are created on the local host server.

I am new to this company so I am not familiar with their infrastructure. I am still a little green with Vsphere.
0
kjs00333Author Commented:
The Datastore its being backed up to is not out of space, but I cannot find the VMDK file it's creating. It seems to be a different name than the VM itself. I browsed the datastore and cannot seem to find it :(
0
kjs00333Author Commented:
I also cannot find any file named QA-Dev Ajax-64Bit-000002.vmdk in the data store when I browse it. If it's not in that VM folder, where would it be located?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
can you screenshot the datastore?
0
kjs00333Author Commented:
Absolutely. Here is what I'm seeing when I browse.

Also, for my other VMs that have the 000001.VMDK file in their data store folders, is it ok to just delete the file?

Thanks again for your help.
datastore-snapshot.jpg
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
No you cannot just manually delete a snapshot you must remove via Snapshot Manager and be patient.

Very odd no snapshots if you create a manual job does it always fail?
0
kjs00333Author Commented:
There are no snapshots showing in the manager, which  is what's confusing me.
Snapshot.jpg
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
test and start another backup?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
kjs00333Author Commented:
I started a new backup of one of the servers in question and it worked. No errors. I have no idea why it did not work as part of the initial job that was already there. However, the other server I have is still not working when I create a new job for it. This one does have the 00001.vmdk file when browsing the datastore. No snapshots exist in the snapshot manager.

This server is throwing up a different error.

Disk "Hard disk 2" has incorrect changed block tracking configuration.

Backing up object "[Admin-DS-12] DocMallSBS11/DocMallSBS11_1.vmdk"

One or more VM disks have incorrect changed block tracking configuration. To resolve this, open VMware vSphere Client, right-click the VM, choose Edit Settings, Options tab, select General, click Configuration Parameters, and set all entries with ‘ctkEnabled’ substring to false. Veeam Backup will then automatically re-enable changed block tracking with the correct settings during the next job run.
BackupDiskFull failed
Client error: Failed to open VDDK disk [[Admin-DS-12] DocMallSBS11/DocMallSBS11_1.vmdk] ( is read-only mode - [true] )
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [admin-vmh-12];Port: 902;Login: [root];VMX Spec: [moref=288];Snapshot mor: [288-snapshot-252];Transports: [nbd]] failed because of the following errors:

Server error: End of file


I have done what the error suggested to do and it still has not corrected the problem. I researched other articles on this error, also followed it's suggestions and still not working. Quite frustrating.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay ensure backup is stopped.

Create a snapshot in Snapshot manager
Wait 60 seconds
Delete all snapshots

Hopefully this will get rid of all snapshots
Then remove all ctk files

Repeat test backup
0
kjs00333Author Commented:
Ok, will do.

Can the CTK files be removed manually when browsing the datastore? Or is there a process for this separately?
0
kjs00333Author Commented:
Turns out making a brand new job in veeam with my VMs in question worked. The other server problem regarding a problem with changed block tracking had to do with hard disk 2 on that specific VM being in independent mode.

Once I turned off independent mode, all was working.
0
kjs00333Author Commented:
The new backup job suggestion ended up fixing the problem for some of the VMs, but the main one in question needed independent mode turned off on that disk.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.