Slow processing rate in Veeam

Dear Experts, we are using Veeam 9.5 to backup our VMs, but the speed of reading disk was very slow, about 20-30 MB/s. The main bottleneck is Source. This problem started happening when we changed to Veeam 9.5 and installed new vCenter appliance. It was fast before, about 100-200 MB/s but with Veeam 8.0 and vCenter Window platform.

The backup proxy is the default Server in Veeam, with settings are like attached picture. How can we increase the speed? Should we change any option?

Environment: VMware vSphere 6.5, 4 Hosts, 24 VMs, Veeam is also a VM, 16 GB RAM, 20 vCPU; 4 Hosts are connected directly to a SAN storage
1.PNG
LVL 5
DP230Network AdministratorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
So you have an

Exchange VM with 40 vCPU
Veeam VM with 20 vCPU

configured ?

have you looked at the performance of your host, and looked at esxtop to check if the VMs are CPU oversubscribbed?

We would like to understand your methodology for assigning vCPU to VMs ?

Also what is the performance of your SAN, e.g. IOPS it was designed to deliver ?
0
DP230Network AdministratorAuthor Commented:
Hi Andrew, yes but they are in different hosts. Attached pic is the result of esxtop command. Could you please check?

I also realized that VMtool has not been installed in VeeamBackup, but we cannot do it since backup is running. Can it be a reason?
esxtop.png
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You should have VMware Tools installed in ALL VMs. It's not optional.
0
Dariusz TykaICT Infrastructure Specialist Senior Commented:
What is your transport mode reported in Veeam backup job log? Hot add, nbd? Also as Andrew mentioned VMware tools should be installed on Veeam machine. Also your backup machine may be overprovisioned. You have 10 concurrent tasks for your Veeam proxy. For each tasks Veeam assignes 1vcpu so I would decrease number of cpus to 10 for test. Also what is your backup repository?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
DP230Network AdministratorAuthor Commented:
Hi Dariusz, is it "nbd"? I found these lines in job log: <transports seq="nbd" nbdCompressionType="none" />

[29.10.2017 01:19:06] <  6724> dt |   Opening live disk source. Disk path: 'vddk://<vddkConnSpec><viConn name="192.168.55.9" authdPort="443" vicPort="443" /><vmxPath vmRef="vm-46" datacenterRef="datacenter-7" datacenterInventoryPath="Datacenter" snapshotRef="snapshot-75" datastoreName="SAN Storage 1" path="XXXXX.com-MX/XXXXX.com-MX.vmx" /><vmdkPath datastoreName="SAN Storage 1" path="XXXXX.com-MX/XXXXX.com-MX.vmdk" /><transports seq="nbd" nbdCompressionType="none" /><readBuffer size="4194304" /></vddkConnSpec>'. ok.
[29.10.2017 01:19:06] <  6724> dt |   Sending notification to the job: disk capacity is '214748364800' bytes.
[29.10.2017 01:19:06] <  6724> dt |   Sending suggested transmission request to target:
[29.10.2017 01:19:06] <  6724> dt |     Suggested block size: [[any]].
[29.10.2017 01:19:06] <  6724> dt |     Is inarguable: [n/a].
[29.10.2017 01:19:06] <  6724> dt |     Is retry: [false].
[29.10.2017 01:19:06] <  6724> dt |   Received approved transmission spec:
[29.10.2017 01:19:06] <  6724> dt |     Block size: [524288].
[29.10.2017 01:19:06] <  6724> dt |     Starting offset: [0].
[29.10.2017 01:19:06] <  6724> dsk|   Initializing vSphere disk changes tracker. Tracker parameters: [SOAP connection: 192.168.55.9;VM (mor): vm-46;Disk key: 2000;Snapshot (mor): snapshot-75;Change ID: *.].
[29.10.2017 01:19:06] <  6724> vim|     Initializing VI SOAP connection...
[29.10.2017 01:19:06] <  6724> vim|       Specification of VI connection
[29.10.2017 01:19:06] <  6724> vim|       {
[29.10.2017 01:19:06] <  6724> vim|         Connection ID: [192.168.55.9].
[29.10.2017 01:19:06] <  6724> vim|         Host: [192.168.55.9].
[29.10.2017 01:19:06] <  6724> vim|         Port: [443].
[29.10.2017 01:19:06] <  6724> vim|         Login: [administrator@YYYY].
[29.10.2017 01:19:06] <  6724> vim|       }
[29.10.2017 01:19:06] <  6724> vim|       [SOAP] Successfully logged in ( server: [192.168.55.9], user: [administrator@YYYY], sessionKey: [52aadd6c-8de4-7540-7ae2-7e3ffaa4a8d3] )
[29.10.2017 01:19:06] <  6724> vim|     VI SOAP connection initialized.
[29.10.2017 01:19:07] <  6724> vim|     CTK_change_extent. Start [0]. Size [133645205504]. Extents number: [2000].
[29.10.2017 01:19:07] <  6724> vim|     {
[29.10.2017 01:19:07] <  6724> vim|       Changes stat.: bytes [118,623,043,584], blocks [226546].
[29.10.2017 01:19:07] <  6724> vim|     }
[29.10.2017 01:19:07] <  6724> dsk|   vSphere CTK tracker initialized.
[29.10.2017 01:19:07] <  6724> dsk|   Initializing swap CTK filter for disk 'VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk'. Std. block size: '524288'.
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 1048576]:\pagefile.sys' not found
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 1048576]:\hiberfil.sys' not found
[29.10.2017 01:19:07] <  6724> dsk|     File '\pagefile.sys' excluded from 'Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 368050176'
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 368050176]:\hiberfil.sys' not found
[29.10.2017 01:19:07] <  6724> dsk| 

Open in new window



 I did not check the VMtool status before running backup, we installed it but have to wait until the end of backup job to restart Veeam server.

The server have 20 vCPU, and Veeam proxy have 10 concurrent tasks, you mean that we should decrease the vCPU of Veeam server? What if we increase the concurrent tasks of Veeam proxy to 20?

The backup repository is NAS device (CIFS) with more than 10 TB free.
0
Dariusz TykaICT Infrastructure Specialist Senior Commented:
Hi 13L@CK_H3@RT,

to be sure pls check in backup log directly in Veeam console. See attached file. It shows SAN but in your case it might be NBD so the slowest transport type. For now I would suggest to install VMware tools on Veeam server and check if that helped. Also can you confim Veeam VM is running on ESXi host with direct access to your SAN?
You can increase number of vCPUs but are your SAN and NAS devices fast enough to deliver/write such amount of concurrent data streams?
 
Veeam.PNG
0
DP230Network AdministratorAuthor Commented:
Hi, after restarted the Veeam server (so VMtools can be installed properly), the processing speed is increased (138 MB/s) but the reading speed of Hard disk is still slow (40 MB/s). Can you please suggest?

Yesterday was Full backup and this one is incremental. Does it change to Hot-Add mode?

incre.PNG
0
Dariusz TykaICT Infrastructure Specialist Senior Commented:
Now it is hotadd transport mode so it is as it should be for Veeam proxy server installed on VM. Regarding read speed - if it was incremental backup then read speed may be lower as it is not sequential read from datastores but random only for data blocks changed since last full backup. You need to monitor next full backup to confirm all is OK.
0
DP230Network AdministratorAuthor Commented:
And now 2 VMs' statuses were stuck at 99%, the snapshots were removed, saving Guestmember.xml were completed but nothing process more. What should we do?
0
Dariusz TykaICT Infrastructure Specialist Senior Commented:
Is it still the issue? But anyhow I think this troubleshooting is beyond the scope of your original question.
0
Dariusz TykaICT Infrastructure Specialist Senior Commented:
Hi 13L@CK_H3@RT
 
do you require any furter assistance?
0
DP230Network AdministratorAuthor Commented:
Hi, the backup speed was increase after we reduced the concurrent tasks in Veeam backup job's settings.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Virtualization

From novice to tech pro — start learning today.