Inconsistent Backup Speeds

I have a really strange issue that I have been trying to track down.  I have a Windows 2008R2 server that is taking over 2 days to backup 3TB of data.  Last week it backed up the data in less than 12 hours over a single gigE connection.  This week it's taking more than 2 days to backup the same data.  

The server is currently in VMWare 3.5.  The data is stored on an EVA 4400 SAN and the ESX hosts connect to the SAN via 4GB Fibre.

Transferring data to from the SAN from clients is ever an issue.  I have both Macs and Windows clients.  The Macs typically use 1-2GB Photoshop files.  I never have a complaint for speed.  

I complain for the backup speeds.  The procedure I use for the backups is staging the data to disk and them backup to tape for off-site storage.

I have a 7TB disk staging area and an HP MSL 6030 LT03 library that is attached to a Windows 2008R2 server.

This week, the backup speeds have dropped to under 700MB/m verses over 4500MB/m in previous weeks.

Looking at my VMWare logs, there is not his ultilization of resources and my Cisco switches are being taxed at all.

I am ata loos for ideas.

Anyone have any input to help me track down the cause?

02/21/2011 12:52:59 PM 170               Total Nodes to be backed up............ 1
02/21/2011 12:52:59 PM 170               Average Throughput: 4,711.52 MB/min
02/21/2011 12:52:59 PM 170               Elapsed Time: 11h 42m 20s
02/21/2011 12:52:59 PM 170               3,097,653.18 MB Written to Media.
LVL 32
nappy_dThere are a 1000 ways to skin the technology cat.Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Have you used your tape cleaning cartridge lately to clean tape heads?
Paul SolovyovskySenior IT AdvisorCommented:
what backup software are you using?  Have you upgraded the firmware and drivers on the autoloader and drive?
Paul SolovyovskySenior IT AdvisorCommented:
Nevermind, didn't see the tag.  If you're using Arcserv drivers change it to HP drivers and see what happens.
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

nappy_dThere are a 1000 ways to skin the technology cat.Author Commented:
I'll give that a try but Arcserve is a b!t*h when it comes to drivers.  The part that is killing me is the D2D staging which is where the process drops in speed, inconsistently.

I put a new cleaning tape in the library last week.

I recently installed HP PSP v8.6

It's a dual tape drive library.  I have two jobs that run.  

Job A' backups up two Mac Final Cut Pro stations.  It does this just fine and backs up the stations over the same gigE connection in under 13 hours for about 3TB of total data.

Job B, backs up VMWare Guest OS's.  This stages the data to a 7TB local volume and then migrates it off to tape for off-site storage.  The migration to tape always runs faily fast and I see speeds upwards of 6000MB/min
Paul SolovyovskySenior IT AdvisorCommented:
I've seen issues with HP Storage Service causing havoc on tape libraries.  Disable all the HP Insight Agents and try the backups.

I've also had many issues with using Arcserve and VM agent.  Which version of ArcServe are you using and are you doing via VM Agent or OS based agent?
nappy_dThere are a 1000 ways to skin the technology cat.Author Commented:
It's Arcserve 12.5sp1

It's being done via the OS Agent as Arcserve does not support VDDK for physical or virtual RDMs.  I would try VCB but I don't have an extra 7TB proxy space.

I will have to let the backup complete in the next 2 days as the estimation is showing that it will take an addit'l 50 hours to complete, which is not normal at all.
Paul SolovyovskySenior IT AdvisorCommented:
Wait until you upgrade to vSphere for VM agent..12.5 is useless. To restore you need to restore from tape then use VMWare Converter to restore back to the ESX hosts, even with 12.5 and VDDK.  I've seen 100GB restore take a full day or more.

Take a granular look and see what's taking so long to backup. Make sure it's not a single entry that slows down to 20MB/s and averages out to 700MB/s once it's done.
nappy_dThere are a 1000 ways to skin the technology cat.Author Commented:
I've done a full restore of VM's testing and real corruptions and it was done in under 40 minutes of an 80GB raw mode backed up guest.

It's just this file backup from a VM that is perplexing.

I'll update more after testing is completed after the backup is done.
nappy_dThere are a 1000 ways to skin the technology cat.Author Commented:
Found the answer. It looks like I had enabled file estimation. This causes the backup to increase in time almost 3 fold as it does a compare from the Arcserve db.

I won't know till the next full backup if this is the fix.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
nappy_dThere are a 1000 ways to skin the technology cat.Author Commented:
Disabling files backup estimation improved thru put.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.