Link to home
Start Free TrialLog in
Avatar of DP230
DP230Flag for United Kingdom of Great Britain and Northern Ireland

asked on

Slow processing rate in Veeam

Dear Experts, we are using Veeam 9.5 to backup our VMs, but the speed of reading disk was very slow, about 20-30 MB/s. The main bottleneck is Source. This problem started happening when we changed to Veeam 9.5 and installed new vCenter appliance. It was fast before, about 100-200 MB/s but with Veeam 8.0 and vCenter Window platform.

The backup proxy is the default Server in Veeam, with settings are like attached picture. How can we increase the speed? Should we change any option?

Environment: VMware vSphere 6.5, 4 Hosts, 24 VMs, Veeam is also a VM, 16 GB RAM, 20 vCPU; 4 Hosts are connected directly to a SAN storage
1.PNG
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

So you have an

Exchange VM with 40 vCPU
Veeam VM with 20 vCPU

configured ?

have you looked at the performance of your host, and looked at esxtop to check if the VMs are CPU oversubscribbed?

We would like to understand your methodology for assigning vCPU to VMs ?

Also what is the performance of your SAN, e.g. IOPS it was designed to deliver ?
Avatar of DP230

ASKER

Hi Andrew, yes but they are in different hosts. Attached pic is the result of esxtop command. Could you please check?

I also realized that VMtool has not been installed in VeeamBackup, but we cannot do it since backup is running. Can it be a reason?
esxtop.png
You should have VMware Tools installed in ALL VMs. It's not optional.
ASKER CERTIFIED SOLUTION
Avatar of Dariusz Tyka
Dariusz Tyka
Flag of Poland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of DP230

ASKER

Hi Dariusz, is it "nbd"? I found these lines in job log: <transports seq="nbd" nbdCompressionType="none" />

[29.10.2017 01:19:06] <  6724> dt |   Opening live disk source. Disk path: 'vddk://<vddkConnSpec><viConn name="192.168.55.9" authdPort="443" vicPort="443" /><vmxPath vmRef="vm-46" datacenterRef="datacenter-7" datacenterInventoryPath="Datacenter" snapshotRef="snapshot-75" datastoreName="SAN Storage 1" path="XXXXX.com-MX/XXXXX.com-MX.vmx" /><vmdkPath datastoreName="SAN Storage 1" path="XXXXX.com-MX/XXXXX.com-MX.vmdk" /><transports seq="nbd" nbdCompressionType="none" /><readBuffer size="4194304" /></vddkConnSpec>'. ok.
[29.10.2017 01:19:06] <  6724> dt |   Sending notification to the job: disk capacity is '214748364800' bytes.
[29.10.2017 01:19:06] <  6724> dt |   Sending suggested transmission request to target:
[29.10.2017 01:19:06] <  6724> dt |     Suggested block size: [[any]].
[29.10.2017 01:19:06] <  6724> dt |     Is inarguable: [n/a].
[29.10.2017 01:19:06] <  6724> dt |     Is retry: [false].
[29.10.2017 01:19:06] <  6724> dt |   Received approved transmission spec:
[29.10.2017 01:19:06] <  6724> dt |     Block size: [524288].
[29.10.2017 01:19:06] <  6724> dt |     Starting offset: [0].
[29.10.2017 01:19:06] <  6724> dsk|   Initializing vSphere disk changes tracker. Tracker parameters: [SOAP connection: 192.168.55.9;VM (mor): vm-46;Disk key: 2000;Snapshot (mor): snapshot-75;Change ID: *.].
[29.10.2017 01:19:06] <  6724> vim|     Initializing VI SOAP connection...
[29.10.2017 01:19:06] <  6724> vim|       Specification of VI connection
[29.10.2017 01:19:06] <  6724> vim|       {
[29.10.2017 01:19:06] <  6724> vim|         Connection ID: [192.168.55.9].
[29.10.2017 01:19:06] <  6724> vim|         Host: [192.168.55.9].
[29.10.2017 01:19:06] <  6724> vim|         Port: [443].
[29.10.2017 01:19:06] <  6724> vim|         Login: [administrator@YYYY].
[29.10.2017 01:19:06] <  6724> vim|       }
[29.10.2017 01:19:06] <  6724> vim|       [SOAP] Successfully logged in ( server: [192.168.55.9], user: [administrator@YYYY], sessionKey: [52aadd6c-8de4-7540-7ae2-7e3ffaa4a8d3] )
[29.10.2017 01:19:06] <  6724> vim|     VI SOAP connection initialized.
[29.10.2017 01:19:07] <  6724> vim|     CTK_change_extent. Start [0]. Size [133645205504]. Extents number: [2000].
[29.10.2017 01:19:07] <  6724> vim|     {
[29.10.2017 01:19:07] <  6724> vim|       Changes stat.: bytes [118,623,043,584], blocks [226546].
[29.10.2017 01:19:07] <  6724> vim|     }
[29.10.2017 01:19:07] <  6724> dsk|   vSphere CTK tracker initialized.
[29.10.2017 01:19:07] <  6724> dsk|   Initializing swap CTK filter for disk 'VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk'. Std. block size: '524288'.
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 1048576]:\pagefile.sys' not found
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 1048576]:\hiberfil.sys' not found
[29.10.2017 01:19:07] <  6724> dsk|     File '\pagefile.sys' excluded from 'Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 368050176'
[29.10.2017 01:19:07] <  6724> dsk|     NTFS object [Disk VDDK:[SAN Storage 1] XXXXX.com-MX/XXXXX.com-MX.vmdk, partition at offset 368050176]:\hiberfil.sys' not found
[29.10.2017 01:19:07] <  6724> dsk| 

Open in new window



 I did not check the VMtool status before running backup, we installed it but have to wait until the end of backup job to restart Veeam server.

The server have 20 vCPU, and Veeam proxy have 10 concurrent tasks, you mean that we should decrease the vCPU of Veeam server? What if we increase the concurrent tasks of Veeam proxy to 20?

The backup repository is NAS device (CIFS) with more than 10 TB free.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of DP230

ASKER

Hi, after restarted the Veeam server (so VMtools can be installed properly), the processing speed is increased (138 MB/s) but the reading speed of Hard disk is still slow (40 MB/s). Can you please suggest?

Yesterday was Full backup and this one is incremental. Does it change to Hot-Add mode?

User generated image
Now it is hotadd transport mode so it is as it should be for Veeam proxy server installed on VM. Regarding read speed - if it was incremental backup then read speed may be lower as it is not sequential read from datastores but random only for data blocks changed since last full backup. You need to monitor next full backup to confirm all is OK.
Avatar of DP230

ASKER

And now 2 VMs' statuses were stuck at 99%, the snapshots were removed, saving Guestmember.xml were completed but nothing process more. What should we do?
Is it still the issue? But anyhow I think this troubleshooting is beyond the scope of your original question.
Hi 13L@CK_H3@RT
 
do you require any furter assistance?
Avatar of DP230

ASKER

Hi, the backup speed was increase after we reduced the concurrent tasks in Veeam backup job's settings.