Avatar of DP230
DP230
Flag for United Kingdom of Great Britain and Northern Ireland asked on

Slow processing rate in Veeam

Dear Experts, we are using Veeam 9.5 to backup our VMs, but the speed of reading disk was very slow, about 20-30 MB/s. The main bottleneck is Source. This problem started happening when we changed to Veeam 9.5 and installed new vCenter appliance. It was fast before, about 100-200 MB/s but with Veeam 8.0 and vCenter Window platform.

The backup proxy is the default Server in Veeam, with settings are like attached picture. How can we increase the speed? Should we change any option?

Environment: VMware vSphere 6.5, 4 Hosts, 24 VMs, Veeam is also a VM, 16 GB RAM, 20 vCPU; 4 Hosts are connected directly to a SAN storage
1.PNG
VirtualizationStorageStorage SoftwareVeeamVMware

Avatar of undefined
Last Comment
DP230

8/22/2022 - Mon
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

So you have an

Exchange VM with 40 vCPU
Veeam VM with 20 vCPU

configured ?

have you looked at the performance of your host, and looked at esxtop to check if the VMs are CPU oversubscribbed?

We would like to understand your methodology for assigning vCPU to VMs ?

Also what is the performance of your SAN, e.g. IOPS it was designed to deliver ?
DP230

ASKER
Hi Andrew, yes but they are in different hosts. Attached pic is the result of esxtop command. Could you please check?

I also realized that VMtool has not been installed in VeeamBackup, but we cannot do it since backup is running. Can it be a reason?
esxtop.png
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

You should have VMware Tools installed in ALL VMs. It's not optional.
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
ASKER CERTIFIED SOLUTION
Dariusz Tyka

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
DP230

ASKER
Hi Dariusz, is it "nbd"? I found these lines in job log: <transports seq="nbd" nbdCompressionType="none" />

[29.10.2017 01:19:06] <  6724> dt |   Opening live disk source. Disk path: 'vddk://<vddkConnSpec><viConn name="192.168.55.9" authdPort="443" vicPort="443" /><vmxPath vmRef="vm-46" datacenterRef="datacenter-7" datacenterInventoryPath="Datacenter" snapshotRef="snapshot-75" datastoreName="SAN Storage 1" path="XXXXX.com-MX/XXXXX.com-MX.vmx" /><vmdkPath datastoreName="SAN Storage 1" path="XXXXX.com-MX/XXXXX.com-MX.vmdk" /><transports seq="nbd" nbdCompressionType="none" /><readBuffer size="4194304" /></vddkConnSpec>'. ok.
[29.10.2017 01:19:06] <  6724> dt |   Sending notification to the job: disk capacity is '214748364800' bytes.
[29.10.2017 01:19:06] <  6724> dt |   Sending suggested transmission request to target:
[29.10.2017 01:19:06] <  6724> dt |     Suggested block size: [[any]].
[29.10.2017 01:19:06] <  6724> dt |     Is inarguable: [n/a].
[29.10.2017 01:19:06] <  6724> dt |     Is retry: [false].
[29.10.2017 01:19:06] <  6724> dt |   Received approved transmission spec:
[29.10.2017 01:19:06] <  6724> dt |     Block size: [524288].
[29.10.2017 01:19:06] <  6724> dt |     Starting offset: [0].
[29.10.2017 01:19:06] <  6724> dsk|   Initializing vSphere disk changes tracker. Tracker parameters: [SOAP connection: 192.168.55.9;VM (mor): vm-46;Disk key: 2000;Snapshot (mor): snapshot-75;Change ID: *.].
[29.10.2017 01:19:06] <  6724> vim|     Initializing VI SOAP connection...
[29.10.2017 01:19:06] <  6724> vim|       Specification of VI connection
[29.10.2017 01:19:06] <  6724> vim|       {
[29.10.2017 01:19:06] <  6724> vim|         Connection ID: [192.168.55.9].
[29.10.2017 01:19:06] <  6724> vim|         Host: [192.168.55.9].
[29.10.2017 01:19:06] <  6724> vim|         Port: [443].
[29.10.2017 01:19:06] <  6724> vim|         Login: [administrator@YYYY].
[29.10.2017 01:19:06] <  6724> vim|       }
[29.10.2017 01:19:06] <  6724> vim|       [SOAP] Successfully logged in ( server: [192.168.55.9], user: [administrator@YYYY], sessionKey: [52aadd6c-8de4-7540-7ae2-7e3ffaa4a8d3] )
[29.10.2017 01:19:06] <  6724> vim|     VI SOAP connection initialized.
[29.10.2017 01:19:07] <  6724> vim|     CTK_change_extent. Start [0]. Size [133645205504]. Extents number: [2000].
[29.10.2017 01:19:07] <  6724> vim|     {
[29.10.2017 01:19:07] <  6724> vim|       Changes stat.: bytes [118,623,043,584], blocks [226546].
[29.10.2017 01:19:07] <  6724> vim|     }
[29.10.2017 01:19:07] <  6724> dsk|   vSphere CTK tracker initialized.
[29.10.2017 01:19:07] <  6724> dsk|   Initializing swap CTK filter for disk 'VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk'. Std. block size: '524288'.
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 1048576]:\pagefile.sys' not found
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 1048576]:\hiberfil.sys' not found
[29.10.2017 01:19:07] <  6724> dsk|     File '\pagefile.sys' excluded from 'Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 368050176'
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 368050176]:\hiberfil.sys' not found
[29.10.2017 01:19:07] <  6724> dsk| 

Open in new window



 I did not check the VMtool status before running backup, we installed it but have to wait until the end of backup job to restart Veeam server.

The server have 20 vCPU, and Veeam proxy have 10 concurrent tasks, you mean that we should decrease the vCPU of Veeam server? What if we increase the concurrent tasks of Veeam proxy to 20?

The backup repository is NAS device (CIFS) with more than 10 TB free.
SOLUTION
Log in to continue reading
Log In
Sign up - Free for 7 days
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
DP230

ASKER
Hi, after restarted the Veeam server (so VMtools can be installed properly), the processing speed is increased (138 MB/s) but the reading speed of Hard disk is still slow (40 MB/s). Can you please suggest?

Yesterday was Full backup and this one is incremental. Does it change to Hot-Add mode?

incre.PNG
Dariusz Tyka

Now it is hotadd transport mode so it is as it should be for Veeam proxy server installed on VM. Regarding read speed - if it was incremental backup then read speed may be lower as it is not sequential read from datastores but random only for data blocks changed since last full backup. You need to monitor next full backup to confirm all is OK.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
DP230

ASKER
And now 2 VMs' statuses were stuck at 99%, the snapshots were removed, saving Guestmember.xml were completed but nothing process more. What should we do?
Dariusz Tyka

Is it still the issue? But anyhow I think this troubleshooting is beyond the scope of your original question.
Dariusz Tyka

Hi 13L@CK_H3@RT
 
do you require any furter assistance?
Your help has saved me hundreds of hours of internet surfing.
fblack61
DP230

ASKER
Hi, the backup speed was increase after we reduced the concurrent tasks in Veeam backup job's settings.