Inconsistent Backup Speeds

Posted on 2011-02-27
Last Modified: 2012-05-11
I have a really strange issue that I have been trying to track down.  I have a Windows 2008R2 server that is taking over 2 days to backup 3TB of data.  Last week it backed up the data in less than 12 hours over a single gigE connection.  This week it's taking more than 2 days to backup the same data.  

The server is currently in VMWare 3.5.  The data is stored on an EVA 4400 SAN and the ESX hosts connect to the SAN via 4GB Fibre.

Transferring data to from the SAN from clients is ever an issue.  I have both Macs and Windows clients.  The Macs typically use 1-2GB Photoshop files.  I never have a complaint for speed.  

I complain for the backup speeds.  The procedure I use for the backups is staging the data to disk and them backup to tape for off-site storage.

I have a 7TB disk staging area and an HP MSL 6030 LT03 library that is attached to a Windows 2008R2 server.

This week, the backup speeds have dropped to under 700MB/m verses over 4500MB/m in previous weeks.

Looking at my VMWare logs, there is not his ultilization of resources and my Cisco switches are being taxed at all.

I am ata loos for ideas.

Anyone have any input to help me track down the cause?

02/21/2011 12:52:59 PM 170               Total Nodes to be backed up............ 1
02/21/2011 12:52:59 PM 170               Average Throughput: 4,711.52 MB/min
02/21/2011 12:52:59 PM 170               Elapsed Time: 11h 42m 20s
02/21/2011 12:52:59 PM 170               3,097,653.18 MB Written to Media.
Question by:nappy_d
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4

Expert Comment

ID: 34993921
Have you used your tape cleaning cartridge lately to clean tape heads?
LVL 42

Expert Comment

ID: 34993931
what backup software are you using?  Have you upgraded the firmware and drivers on the autoloader and drive?
LVL 42

Expert Comment

ID: 34993932
Nevermind, didn't see the tag.  If you're using Arcserv drivers change it to HP drivers and see what happens.
The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

LVL 32

Author Comment

ID: 34994094
I'll give that a try but Arcserve is a b!t*h when it comes to drivers.  The part that is killing me is the D2D staging which is where the process drops in speed, inconsistently.

I put a new cleaning tape in the library last week.

I recently installed HP PSP v8.6

It's a dual tape drive library.  I have two jobs that run.  

Job A' backups up two Mac Final Cut Pro stations.  It does this just fine and backs up the stations over the same gigE connection in under 13 hours for about 3TB of total data.

Job B, backs up VMWare Guest OS's.  This stages the data to a 7TB local volume and then migrates it off to tape for off-site storage.  The migration to tape always runs faily fast and I see speeds upwards of 6000MB/min
LVL 42

Expert Comment

ID: 34994115
I've seen issues with HP Storage Service causing havoc on tape libraries.  Disable all the HP Insight Agents and try the backups.

I've also had many issues with using Arcserve and VM agent.  Which version of ArcServe are you using and are you doing via VM Agent or OS based agent?
LVL 32

Author Comment

ID: 34994133
It's Arcserve 12.5sp1

It's being done via the OS Agent as Arcserve does not support VDDK for physical or virtual RDMs.  I would try VCB but I don't have an extra 7TB proxy space.

I will have to let the backup complete in the next 2 days as the estimation is showing that it will take an addit'l 50 hours to complete, which is not normal at all.
LVL 42

Expert Comment

ID: 34994226
Wait until you upgrade to vSphere for VM agent..12.5 is useless. To restore you need to restore from tape then use VMWare Converter to restore back to the ESX hosts, even with 12.5 and VDDK.  I've seen 100GB restore take a full day or more.

Take a granular look and see what's taking so long to backup. Make sure it's not a single entry that slows down to 20MB/s and averages out to 700MB/s once it's done.
LVL 32

Author Comment

ID: 34994280
I've done a full restore of VM's testing and real corruptions and it was done in under 40 minutes of an 80GB raw mode backed up guest.

It's just this file backup from a VM that is perplexing.

I'll update more after testing is completed after the backup is done.
LVL 32

Accepted Solution

nappy_d earned 0 total points
ID: 34997389
Found the answer. It looks like I had enabled file estimation. This causes the backup to increase in time almost 3 fold as it does a compare from the Arcserve db.

I won't know till the next full backup if this is the fix.
LVL 32

Author Closing Comment

ID: 35120564
Disabling files backup estimation improved thru put.

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
Microservice architecture adoption brings many advantages, but can add intricacy. Selecting the right orchestration tool is most important for business specific needs.
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
This tutorial will show how to configure a single USB drive with a separate folder for each day of the week. This will allow each of the backups to be kept separate preventing the previous day’s backup from being overwritten. The USB drive must be s…

623 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question