• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 425
  • Last Modified:

Please need help to investigate a problem I am having with slow Veeam Backups.

Good Day Everyone

I please need help to investigate a problem we are having with Veeam Backups.

We running a new install of Veeam 9 and have a backup speed issue on production HP DL380e G8’s. The production hosts are running ESXi 6. Backup speeds we getting is 40MB/s odd. The backups run to a QNAP NAS device connected via iSCSI .

Not to play around with the production environment too much, we setup a test environment with the below;
1 i3 Desktop running VMware ESXi 5.1
1 HP DL380 G5 running VMware ESXi 5.1

Both are being backed up with the same physical Veeam 9 backup and replication server.
Target is a QNAP NAS connect via iSCSI to the Veeam Backup server.
Each ESXI host has one Veeam Proxy VM and one test computer VM for backing up.
The Test and Proxy VM's are stored on the local storage of both hosts. ie the local disks
All connected to same switch in an isolated network all NIC's connected at 1000Mbps.

Here is the thing.
Backing up the VM from the Desktop ESXI we get ~80MB/s
Backing up the VM from the DL380 G5 ESXI we get ~40MB/s
hotadd working on both.

So on the test environment we also get slow backup speed of around 40MB\35MB/s as we do with the production environment.

Veeam support suggested that we run the linux dd command to test read speed. The below is what i get.

From Test Desktop ESXI host
~ # time dd if="/vmfs/volumes/58079b57-d19adb34-f141-dc4a3e3c8268/Test VM/Test VM-flat.vmdk" of=/dev/null bs=1M count=3072
3072+0 records in
3072+0 records out
real    0m 25.43s
user    0m 3.80s
sys     0m 0.00s
~ #

From test DL380 G5 ESXI host
[root@localhost:~] time dd if="/vmfs/volumes/581c8a5f-d2b5977b-3cb0-00215aebb35e/Test Machine 1/Test Machine 1-flat.vmdk" of=/dev/null bs=1M count=3072
3072+0 records in
3072+0 records out
real    0m 50.90s
user    0m 6.79s
sys     0m 0.00s
[root@localhost:~]

From Production DL380e G8 ESXI host
[root@vmhost02:~] time dd if="/vmfs/volumes/4cb60574-39ae86c1-fc5a-002655f4a5e9/TDM Console/TDM Console-flat.vmdk" of=/dev/null bs=1M count=3072
3072+0 records in
3072+0 records out
real    0m 53.95s
user    0m 2.24s
sys     0m 0.00s
[root@vmhost02:~]

Why would the read speed be so much better on the little test desktop ESXI host? If we could understand this then perhaps we know what needs to be looked at on the production servers to get better backup performance. I would think that on the production servers we should also get around ~80MB/s as we would with the test desktop host? Unless i am not understanding or missing something.

Thank you.
0
iamdieter
Asked:
iamdieter
  • 8
  • 8
  • 3
2 Solutions
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you have BBWC (Battery Backup Write Cache) on the Smart Array Storage controller ?

what storage is in the Desktop compared to the ESXi hosts ?

RAID versus Single Disk ?
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi,

How did you present the iSCSI to the Veeam Windows server? What type of iSCSI Volumes did you created  in the QNAP?

In your Veeam Server install CristalDiskMask and run on the Disks that are mounted in the Windows. Check what is the speed that you can save on those repository.

You get  ~80MB/s and ~40MB/s, but you should get more information in the Veeam Backup details(like read / write, the bottleneck etc.)
0
 
iamdieterAuthor Commented:
Thank you for the replies

The desktop has a standard single SATA Drive

The test DL380 G5 currently has a single 10k 150GB SAS drive in a RAID0 Array. We do not have another disk to put in the Array. The write cache battery is faulty causing the write cache accelerator to be disabled.

The productions DL380e G8 are connected via fibre to a DELL SAN. The volume we testing on, is across Tier 1 and Tier 3.  Tier 1 RAID 10 10k disks, and Tier 3 RAID 5 7.2k disks. Write Cache is enabled.


The QNAP is connected directly to the physical Veeam Backup Server using a single crossover Ethernet cable connecting at 1000Mbps.
Both backup jobs target the same volume on the QNAP. Ie the same repository is used for both.

Using CrystalDiskMark, running on the Physical Veeam Backup Server
 
Test Count 5, 1GIB, C: (Local Drive of server)

Seq Q32T1 Read 87.34 MB/s  Write 87.48 MB/s
4K Q32T1 Read 2.31 MB/s  Write 1.72 MB/s
Seq Read 89.34 MB/s  Write 58.93 MB/s
4K Read 0.77 MB/s  Write 0.68 MB/s

Test Count 5, 1GIB, G: (Volume on QNAP)

Seq Q32T1 Read 112.1 MB/s  Write 118.3 MB/s  
4K Q32T1 Read 105.8 MB/s  Write 87.34 MB/s  
Seq Read 105.5 MB/s  Write 110.3 MB/s  
4K Read 16.73 MB/s  Write 16.32 MB/s  

The DL380e G8 reports bottlenecks as Source 99% Proxy 18%, Network 3% Target 0%.
The desktop job reports bottlenecks as Source 91%, Proxy 37%, Network 27%, Target 0%.

The volume info on the QNAP.
LUN Type: Block-Based
Sector Size 512 bytes
0
Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If the Cache is faulty, replace the battery, this will improve your performance.

and enable as follows

25% Read / 75% Write
0
 
iamdieterAuthor Commented:
I agree,

But the very odd thing is we getting low performance on the production DL380 G8e as well.
These are connect via Fiber to a SAN?

Just to reiterate.

Test Desktop Speed (local SATA disk) - Speed at 80MB/s
Production DL380 G8 (Connect to DELL SAN via Fibre) - Speed at 40MB/s

Test DL380 G5 (Local Disk) Speed 40MB/s -  lets take this server out the equation

Problem seems to be reading the VMDK from the data stores with the production DL380e G8
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi,

As you see both says first bottleneck is the source(the VMware environment and your Datastores). Even this is not 100% acquired.

Secondly can you explain a little better your Production VMware environment?? How many ESXi hosts(is ESXi 5.1)? vCenter??
In the Veeam configuration you are connecting directly to the ESXi hosts, or to the vCenter?

You are using Veeam Proxys?? Same subnet / VLAN? Different subnets/VLANs?
0
 
iamdieterAuthor Commented:
I agree the bottleneck seems to be the source and it is almost as if Veeam is struggling to read the VMDK's from the datastore in the production environment.  

Did a quick diagram to help show our production environment.
Veeam 9 has vCenter added.

Veeam-Backup-Dia.PNG
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi,

Why you have 2 Veeam Proxys for such a small environment? Are those ESXi hosts running in different environments(subnets/vlans)?

Veeam proxy is use or to release resources from the main Veeam Backup Server, or use to run in a different subnet/VLAN so that Veeam Backup server can connect without any network disturbances. For such a small environment I don't see this is the case.
0
 
iamdieterAuthor Commented:
All Physical and Virtual machines are on the same network same vLan.
ie all on the default data vLan. 10.8.64.0/22 Both Hosts are the same environment.

I tried running a job using the Physical Veeam Backup Server as the proxy, the backup fails with error "No proxy is able to process this VM due to proxy processing restrictions". (with the proxy being set to use only hotadd).

I used a virtual proxy per host to ensure that hotadd works in my tests.

Thank you for the info, ill look at running one proxy.
Please confirm, i read somewhere proxy's must be virtual to work or is a physical proxy meant to work?

However i don't think to many proxy's could be causing the slow performance or could it?
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi,

Veeam Backup Server will and should create an embedded proxy. For your environment no need to have extra proxy's.

When you get that error, you have many things that can trigger that. Check here: https://www.veeam.com/kb1857

In you case, most likely is cannot connect to the vCenter/ESXi datastores.
0
 
iamdieterAuthor Commented:
Thank you for the info and will look into it.  

When hotadd works we still get the slow performane. Any idea what this could be?

Looking at the dd command results still suggests a read performance problem.  Would you agree?
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi,

Take a look at this article that I have wrote regarding Veeam and Proxy.

https://www.experts-exchange.com/articles/26501/Veeam-How-to-enable-Direct-NFS-Access-backup-Access-feature.html

Veeam Server and/or Veeam Proxy, for good performance need always to have direct connection(or work in the same network) of the source Storage. This is why we use proxy to provide that.
1
 
iamdieterAuthor Commented:
Thank you for the info on Direct Access and will have a look.

Still would really like to understand why the hotadd mode gets half the speed on the DL380e G8 vs the desktop with hotadd.
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi

Do they use the same proxy for that backup test?

Are they both physical?

If not, just config to use always the same proxy and test.
0
 
iamdieterAuthor Commented:
They dont use the same proxy.

The Desktop is its own ESXI host running 5.1.
Has a test VM and has a Proxy.
Desktop is not on the same SAN, it is using a local disk.

The production environment is using the proxys as discussed above.

hotadd is working on the Dekstop Backup Jobs and hotadd is working on the production.

Desktop speed ~80MB/s
Production speed ~40MB/s
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi

If not backup the same VMs, not using the same proxy, not the same SAN, so you cannot compare them.
0
 
iamdieterAuthor Commented:
Hi

We managed to get direct access to work on the fiber.
Thanks for you help.

Still a mystery why a little Desktop would performance twice as fast as a SAN environment with hotadd.
0
 
Luciano PatrãoICT Senior Infraestructure Engineer Commented:
Hi

I am glad that did help.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
when you fix your BBWC, report back, what performance is like!
0

Featured Post

Configuration Guide and Best Practices

Read the guide to learn how to orchestrate Data ONTAP, create application-consistent backups and enable fast recovery from NetApp storage snapshots. Version 9.5 also contains performance and scalability enhancements to meet the needs of the largest enterprise environments.

  • 8
  • 8
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now