?
Solved

Backup Exec 2012 + slow through/low nic util %

Posted on 2013-06-13
12
Medium Priority
?
987 Views
Last Modified: 2013-07-03
Running Backup Exec 2012 on Server 2008 R2 with an iSCSI NAS for backup job disc storage.

This server is in a different site, therefore I haven't had to touch it in a while, but the tech in that site reported that backup jobs have been dragging lately in terms of elapsed time.. in particular Exchange 2010.

I remote into both servers and right off the bat I notice how nic utilization on either server never exceeds 21.6%

Is there a reason for this? Is there any way to remove this apparent limit?
0
Comment
Question by:Ben Hart
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 3
12 Comments
 
LVL 44

Expert Comment

by:Amit
ID: 39245759
Best practice, is to avoid backup during production hours, also if server is not recycled from long time, you can reboot the Exchange server and check if that improves the performance. Also, it depends on DB size. If db size is high, backup process can be slow.
0
 
LVL 14

Author Comment

by:Ben Hart
ID: 39245772
*sigh*

Both my backup source servers and the BUE server itself all have secondary 'backup' nics that all are cabled via their own Cisco layer 2 switch.

I thank you for offering some suggestions but you rather ignored my real question there bucko.
0
 
LVL 63

Expert Comment

by:Simon Butler (Sembee)
ID: 39248250
What are you backing up?
Are you doing an information store backup or a GRT/Brick level backup?

The usual bottleneck is storage, in many cases the NIC can outrun the storage controller. Therefore the first thing I would do is run some perfmon counters when the backup is running to see whether you are getting storage peaks.

Simon.
0
Office 365 Training for Admins - 7 Day Trial

Learn how to provision tenants, synchronize on-premise Active Directory, implement Single Sign-On, customize Office deployment, and protect your organization with eDiscovery and DLP policies.  Only from Platform Scholar.

 
LVL 14

Author Comment

by:Ben Hart
ID: 39248297
Normally I'd whole-heartedly agree Simon.  But this is a recent development.. like last winter the full Exchange job was completing in under 12 hours for a total of ~600gb.  Yesterday I was watching it, 17 hours and 658gb later it was still processing.

But my real question is still missed, I'm wondering more about the observed nic throughput limit.
0
 
LVL 63

Expert Comment

by:Simon Butler (Sembee)
ID: 39248406
It could still be a storage issue - if you have a hard disk failing or something like that.
Have you tried sending other traffic through the NIC to see if you can flood it? You need to eliminate the application layer. If you can max out the NIC with a regular file copy (for example) then that points at the application/s being the source.

Simon.
0
 
LVL 14

Author Comment

by:Ben Hart
ID: 39260398
I cannot max out the nic with a normal file copy.. either using windows or a util like RoboCopy.  Nothing I do exceeds 30% of the reported traffic.
0
 
LVL 14

Author Comment

by:Ben Hart
ID: 39260697
Actually here's a monkey wrench that I didnt notice before.. starting the backup job I'm watching the 'backup' nics on both the server and the source box being backed up.  Both nics eceed 75% utilization for about 3 minutes.  Then both drop to under 25% and fluctuate between 0 and 25 but never again going above that.

I have jumbo frames enabled on both nics and teh switch between them, I have RSS enabled on both ends, and flow control on both.  Nic drivers on both servers are less than 2 years old, IDK what else to try.

I even disabled GRT on this backup job.. has made zero difference.
0
 
LVL 14

Author Comment

by:Ben Hart
ID: 39260720
i realize that there's many.. many things that affect a backup job and network throughput.  I have updated nic drivers, I've replaced patch cords with Cat6, tests the disk subsystem performance on both the source server and the backup servers volumes.  If the theoretical maximum throughput on gigabit Ethernet is 125MB/s or 7500MB/m.. my source servers disk perf is @358MB/s on Read.  I'd have to assume the bottleneck would be the network.

Honestly I've found no way to test writing on the iscsi nas the BUE server uses because the utility wanted it to be blank.

In any event I posted here as a last resort...
0
 
LVL 63

Expert Comment

by:Simon Butler (Sembee)
ID: 39260767
Last time I saw something like this it was Jumbo Frames that was the cause of the problem. Switch I think it was. I also have a distant recollection of AV software causing throttling in certain scenarios.
The fact that it happens even on a straight file copy rules out Backup Exec as the cause, it is something with the underlying system.

Can you get any faster speeds from another source system to the same target?

Simon.
0
 
LVL 14

Author Comment

by:Ben Hart
ID: 39260789
Actually JF wasn't enabled until yesterday during my testing.  I enabled it not remembering if I had had it enabled two years ago when I set this system up.  (insider hint: another tech changed everything around< so now I'm having to fix it)

Also I do not run AV on my Exchange server, but I do on my BUE server.

two years ago I had setup all teh source servers and teh BUE box with secondary or tertiary nics, all addresses on a 192 subnet and all cabled to a separate Cisco Small business gigabit switch.  This other tech, recabled the entire mess routing them all through the core switch on it's own VLAN.  This was done months ago and to my knowledge it's been slow and getting slower ever since.  My test today was putting BUE and Exchange back onto that Small Business switch.  Which did seem to help initially.. I had throughput rates of 4K+ MB/minute in Backup Exec, but after 3-4 minute it tapers off to below 1000 and keeps gradually falling to sub-600's after roughly 20 minutes.

I agree with you that it's likely not BUE itself causing this.
0
 
LVL 14

Accepted Solution

by:
Ben Hart earned 0 total points
ID: 39284162
I guess Simons out of ideas too.  Ok so I guess my issue is sort-of resolved.

Between all of my Exchange data we're backing up almost 900GB and using GRT the job is so badly slow.. we're talking an overall process speed of less than 700MB/min.  Without GRT it runs upwards of 1600MB/min, so maybe that's by design... but I was able to get a tiny bit more throughput by disabling all forms of nic offloading, flow control, ethernet@wirespeed and hard setting the speed/duplex.

We're actually looking at moving to Exchange Online Archiving so I can enforce retention policies and shrink our DB sizes down.
0
 
LVL 14

Author Closing Comment

by:Ben Hart
ID: 39295975
Simon was'nt able to offer a clear solution.  While my end result and how I got there didn't directly address my problem.. it is what I've come to as a resolution, as it were.
0

Featured Post

Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Unified and professional email signatures help maintain a consistent company brand image to the outside world. This article shows how to create an email signature in Exchange Server 2010 using a transport rule and how to overcome native limitations …
Resolving an irritating Remote Desktop connection that stops your saved credentials from being used.
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…
Suggested Courses

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question