Solved

Please need help to investigate a problem I am having with slow Veeam Backups.

Posted on 2016-11-08
19
71 Views
Last Modified: 2016-11-10
Good Day Everyone

I please need help to investigate a problem we are having with Veeam Backups.

We running a new install of Veeam 9 and have a backup speed issue on production HP DL380e G8’s. The production hosts are running ESXi 6. Backup speeds we getting is 40MB/s odd. The backups run to a QNAP NAS device connected via iSCSI .

Not to play around with the production environment too much, we setup a test environment with the below;
1 i3 Desktop running VMware ESXi 5.1
1 HP DL380 G5 running VMware ESXi 5.1

Both are being backed up with the same physical Veeam 9 backup and replication server.
Target is a QNAP NAS connect via iSCSI to the Veeam Backup server.
Each ESXI host has one Veeam Proxy VM and one test computer VM for backing up.
The Test and Proxy VM's are stored on the local storage of both hosts. ie the local disks
All connected to same switch in an isolated network all NIC's connected at 1000Mbps.

Here is the thing.
Backing up the VM from the Desktop ESXI we get ~80MB/s
Backing up the VM from the DL380 G5 ESXI we get ~40MB/s
hotadd working on both.

So on the test environment we also get slow backup speed of around 40MB\35MB/s as we do with the production environment.

Veeam support suggested that we run the linux dd command to test read speed. The below is what i get.

From Test Desktop ESXI host
~ # time dd if="/vmfs/volumes/58079b57-d19adb34-f141-dc4a3e3c8268/Test VM/Test VM-flat.vmdk" of=/dev/null bs=1M count=3072
3072+0 records in
3072+0 records out
real    0m 25.43s
user    0m 3.80s
sys     0m 0.00s
~ #

From test DL380 G5 ESXI host
[root@localhost:~] time dd if="/vmfs/volumes/581c8a5f-d2b5977b-3cb0-00215aebb35e/Test Machine 1/Test Machine 1-flat.vmdk" of=/dev/null bs=1M count=3072
3072+0 records in
3072+0 records out
real    0m 50.90s
user    0m 6.79s
sys     0m 0.00s
[root@localhost:~]

From Production DL380e G8 ESXI host
[root@vmhost02:~] time dd if="/vmfs/volumes/4cb60574-39ae86c1-fc5a-002655f4a5e9/TDM Console/TDM Console-flat.vmdk" of=/dev/null bs=1M count=3072
3072+0 records in
3072+0 records out
real    0m 53.95s
user    0m 2.24s
sys     0m 0.00s
[root@vmhost02:~]

Why would the read speed be so much better on the little test desktop ESXI host? If we could understand this then perhaps we know what needs to be looked at on the production servers to get better backup performance. I would think that on the production servers we should also get around ~80MB/s as we would with the test desktop host? Unless i am not understanding or missing something.

Thank you.
0
Comment
Question by:iamdieter
  • 8
  • 8
  • 3
19 Comments
 
LVL 117
ID: 41878889
do you have BBWC (Battery Backup Write Cache) on the Smart Array Storage controller ?

what storage is in the Desktop compared to the ESXi hosts ?

RAID versus Single Disk ?
0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 41880328
Hi,

How did you present the iSCSI to the Veeam Windows server? What type of iSCSI Volumes did you created  in the QNAP?

In your Veeam Server install CristalDiskMask and run on the Disks that are mounted in the Windows. Check what is the speed that you can save on those repository.

You get  ~80MB/s and ~40MB/s, but you should get more information in the Veeam Backup details(like read / write, the bottleneck etc.)
0
 

Author Comment

by:iamdieter
ID: 41880398
Thank you for the replies

The desktop has a standard single SATA Drive

The test DL380 G5 currently has a single 10k 150GB SAS drive in a RAID0 Array. We do not have another disk to put in the Array. The write cache battery is faulty causing the write cache accelerator to be disabled.

The productions DL380e G8 are connected via fibre to a DELL SAN. The volume we testing on, is across Tier 1 and Tier 3.  Tier 1 RAID 10 10k disks, and Tier 3 RAID 5 7.2k disks. Write Cache is enabled.


The QNAP is connected directly to the physical Veeam Backup Server using a single crossover Ethernet cable connecting at 1000Mbps.
Both backup jobs target the same volume on the QNAP. Ie the same repository is used for both.

Using CrystalDiskMark, running on the Physical Veeam Backup Server
 
Test Count 5, 1GIB, C: (Local Drive of server)

Seq Q32T1 Read 87.34 MB/s  Write 87.48 MB/s
4K Q32T1 Read 2.31 MB/s  Write 1.72 MB/s
Seq Read 89.34 MB/s  Write 58.93 MB/s
4K Read 0.77 MB/s  Write 0.68 MB/s

Test Count 5, 1GIB, G: (Volume on QNAP)

Seq Q32T1 Read 112.1 MB/s  Write 118.3 MB/s  
4K Q32T1 Read 105.8 MB/s  Write 87.34 MB/s  
Seq Read 105.5 MB/s  Write 110.3 MB/s  
4K Read 16.73 MB/s  Write 16.32 MB/s  

The DL380e G8 reports bottlenecks as Source 99% Proxy 18%, Network 3% Target 0%.
The desktop job reports bottlenecks as Source 91%, Proxy 37%, Network 27%, Target 0%.

The volume info on the QNAP.
LUN Type: Block-Based
Sector Size 512 bytes
0
 
LVL 117
ID: 41880408
If the Cache is faulty, replace the battery, this will improve your performance.

and enable as follows

25% Read / 75% Write
0
 

Author Comment

by:iamdieter
ID: 41880411
I agree,

But the very odd thing is we getting low performance on the production DL380 G8e as well.
These are connect via Fiber to a SAN?

Just to reiterate.

Test Desktop Speed (local SATA disk) - Speed at 80MB/s
Production DL380 G8 (Connect to DELL SAN via Fibre) - Speed at 40MB/s

Test DL380 G5 (Local Disk) Speed 40MB/s -  lets take this server out the equation

Problem seems to be reading the VMDK from the data stores with the production DL380e G8
0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 41880450
Hi,

As you see both says first bottleneck is the source(the VMware environment and your Datastores). Even this is not 100% acquired.

Secondly can you explain a little better your Production VMware environment?? How many ESXi hosts(is ESXi 5.1)? vCenter??
In the Veeam configuration you are connecting directly to the ESXi hosts, or to the vCenter?

You are using Veeam Proxys?? Same subnet / VLAN? Different subnets/VLANs?
0
 

Author Comment

by:iamdieter
ID: 41880651
I agree the bottleneck seems to be the source and it is almost as if Veeam is struggling to read the VMDK's from the datastore in the production environment.  

Did a quick diagram to help show our production environment.
Veeam 9 has vCenter added.

Veeam-Backup-Dia.PNG
0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 41880686
Hi,

Why you have 2 Veeam Proxys for such a small environment? Are those ESXi hosts running in different environments(subnets/vlans)?

Veeam proxy is use or to release resources from the main Veeam Backup Server, or use to run in a different subnet/VLAN so that Veeam Backup server can connect without any network disturbances. For such a small environment I don't see this is the case.
0
 

Author Comment

by:iamdieter
ID: 41880822
All Physical and Virtual machines are on the same network same vLan.
ie all on the default data vLan. 10.8.64.0/22 Both Hosts are the same environment.

I tried running a job using the Physical Veeam Backup Server as the proxy, the backup fails with error "No proxy is able to process this VM due to proxy processing restrictions". (with the proxy being set to use only hotadd).

I used a virtual proxy per host to ensure that hotadd works in my tests.

Thank you for the info, ill look at running one proxy.
Please confirm, i read somewhere proxy's must be virtual to work or is a physical proxy meant to work?

However i don't think to many proxy's could be causing the slow performance or could it?
0
What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

 
LVL 22

Assisted Solution

by:Luciano Patrão
Luciano Patrão earned 500 total points
ID: 41880916
Hi,

Veeam Backup Server will and should create an embedded proxy. For your environment no need to have extra proxy's.

When you get that error, you have many things that can trigger that. Check here: https://www.veeam.com/kb1857

In you case, most likely is cannot connect to the vCenter/ESXi datastores.
0
 

Author Comment

by:iamdieter
ID: 41880922
Thank you for the info and will look into it.  

When hotadd works we still get the slow performane. Any idea what this could be?

Looking at the dd command results still suggests a read performance problem.  Would you agree?
0
 
LVL 22

Accepted Solution

by:
Luciano Patrão earned 500 total points
ID: 41880949
Hi,

Take a look at this article that I have wrote regarding Veeam and Proxy.

https://www.experts-exchange.com/articles/26501/Veeam-How-to-enable-Direct-NFS-Access-backup-Access-feature.html

Veeam Server and/or Veeam Proxy, for good performance need always to have direct connection(or work in the same network) of the source Storage. This is why we use proxy to provide that.
1
 

Author Comment

by:iamdieter
ID: 41881606
Thank you for the info on Direct Access and will have a look.

Still would really like to understand why the hotadd mode gets half the speed on the DL380e G8 vs the desktop with hotadd.
0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 41881718
Hi

Do they use the same proxy for that backup test?

Are they both physical?

If not, just config to use always the same proxy and test.
0
 

Author Comment

by:iamdieter
ID: 41881866
They dont use the same proxy.

The Desktop is its own ESXI host running 5.1.
Has a test VM and has a Proxy.
Desktop is not on the same SAN, it is using a local disk.

The production environment is using the proxys as discussed above.

hotadd is working on the Dekstop Backup Jobs and hotadd is working on the production.

Desktop speed ~80MB/s
Production speed ~40MB/s
0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 41881935
Hi

If not backup the same VMs, not using the same proxy, not the same SAN, so you cannot compare them.
0
 

Author Comment

by:iamdieter
ID: 41882180
Hi

We managed to get direct access to work on the fiber.
Thanks for you help.

Still a mystery why a little Desktop would performance twice as fast as a SAN environment with hotadd.
0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 41882195
Hi

I am glad that did help.
0
 
LVL 117
ID: 41882227
when you fix your BBWC, report back, what performance is like!
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Join & Write a Comment

This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
In this article, I show you step by step with screenshots to assist you - HOW TO: Deploy and Install the VMware vCenter Server Appliance 6.5 (VCSA 6.5), with some helpful tips along the way.
Teach the user how to install ESXi 5.5 and configure the management network System Requirements: ESXi Installation:  Management Network Configuration: Management Network Testing:
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now