backup exec 12.5 remote agent utility slow job rate until reboot

I'm having an issue with slow backup job rate on an exchange server with the remote agent utility.

Server A is a 2008 file server with Symantec Backup Exec 12.5 with an IBM Ultrium-HH3 Tape Drive.  Server B is an exchange 2010 server on server 2008 with the Symantec Backup Exec remote agent utility, version 12.5.  The issue here is that every day, the job rate continues to decline.  If I reboot the exchange server (server B) the job rate will be normal, and the backup will complete in 3 hours.  Each day, the job rate declines and the backup time increases by several hours, to where eventually, the job will go so slow that it will last 24 hours.  Of course, we can temporarily resolve this by rebooting the exchange server, which starts the process all over again.

I have temporarily resolved the issue by scheduling a reboot of the exchange server every 3 days; however, I'd like to find a better fix than that.

 notice the job rate declines each day until a reboot
When I look at the job history, I can see that the job slows down on the system state of the exchange server, which uses the remote agent).  The first day after a reboot, the exchange system state will run around 350mb/minute.  Each day, it decreases by a considerable amount, and it will eventually get lower than 30mb/minute. Every other backup set runs fine and quickly each day.

 job rate on remote agent's system state slows down more each day.
I have ran all live updates with Symantec and have re-installed the remote agent on the exchange server (server B)

Thank you in advance for any help or suggestions
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Have you checked the Exchange server to make sure no other background processes are running at the same time the  backup is running? Are you using the Backup Exec Exchange utility/add on?

 I have had similar issues in the past with Backup Exec on a computer using the remote agent.  Secondly because the backup is running over the network I would make sure you have the most current drivers for the NIC on the Exchange Server.
natecoyeAuthor Commented:
Yes, I have checked processes on the exchange server. The big on there is store.exe, which I've heard uses as much ram as it can get.  I originally thought that was the issue, because when you reboot that server, store.exe doesn't start using as much ram, but continually uses more ram over time.  We also have Trend worry-free business manager on there.  I've unloaded the client on both machines, but it had little effect.

I do not have the backup exec exchange utility, but I also wonder if it's even an exchange issue when the problem occurs when backing up the system state.

I have ran all updates on both servers, including the most recent exchange updates.  Also, both servers are connected to the same switch and are connected at 1gig.
So for testing purposes if you remove the System state from the backup job does it always run quickly even without a reboot? The system state should be the fastest part of the backup.

I am assuming you have checked the event logs on the Exchange server to verify you are not getting drive or controller errors?

Not that this post solves the problem but it appears other users are having the same issue with the system state in version 12.5 on 2008 server.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
SolarWinds® IP Control Bundle (IPCB)

Combines SolarWinds IP Address Manager and User Device Tracker to help detect IP conflicts, quickly identify affected systems, and help your team take near instantaneous action. Help improve visibility and enhance reliability with SolarWinds IP Control Bundle.

natecoyeAuthor Commented:
I checked the event log on both machines, and they're clean.

I forgot to mention, but yes, I have looked at the article you posted.  I'm discouraged that it says "There are no plans to address this issue by way of a patch or hotfix in the current or previous versions of the software at the present time."  Also, the article states that Backup Exec 2010 and 2010R2 are affected, so It doesn't seem that upgrading to 2010 will help.  I may just have to stick with my scheduled task to reboot twice a week.

I can try to remove the system state for a few days, but I'm concerned that I would need it in a restore scenario.  
I am experiencing same issues with Backup Exec 2010 R2 not only on Exchange server but on our file servers randomly. It happens about once every 2 weeks or so. I have not been able find a solution.

I did find that my backup jobs slow down when BackupExec database maintenance is performed. I don't know if it is related but worth of checking out...
natecoyeAuthor Commented:
I hadn't thought of database maintenance.  I moved the maintenance time back to where it wouldn't conflict with my scheduled backup.  It's worth a shot, I guess.
Something I always check when working on a BackupExec server and slow backups to tape is to make sure that the BackupExec server is running the most up to date drivers for the tape drive from Symantec. For me 9 out 10 times this takes care of the issue. It may not be the case here but if you haven't checked into it yet its something to think about.
natecoyeAuthor Commented:
also a good idea.  I verified that I have the current Symantec drivers for the IBM ULTRIM-HH3
Guillermo FeijóoSystems administratorCommented:
Have you tried unchecking the software compressión, (using hardware compression instead), and doing a database compactation in order to increase performance of that job?

Worked for me, hope it helps you.
natecoyeAuthor Commented:
it was set on Hardware [if available, otherwise software], so I set it to Hardware [if available, otherwise none].  Also, I ran the beutility.exe and did a database repair, compact and check for consistency.  We'll see if any of that helps.  Thanks
natecoyeAuthor Commented:
After trying all of these ideas, I awarded points to Patmac951, because I think the link he posted was the most helpful.  It looks like Symantec is aware of the issue and has no plans to resolve it in any current version of the software.  For now, I will schedule a reboot of the remote server twice a week as a work-around.
I know it's after the fact; but I always run a batch job to terminate the local AV scanning when backing up a server as it used to take AGES.  Of course, I run a post backup job to restart the AV service.
natecoyeAuthor Commented:
I did think about doing that, but trend doesn't allow command-line access to shut down the services, so I wasn't sure how to go about doing that.  Good idea, though.  When I manually unloaded the AV, the backup did go faster, but I still had the same result of the backups getting slower over a few days, so I don't think it was an AV issue.
In the batch file just do a net stop servicename and then restart post job completion.  We use Sophos which also doesn't allow you to stop; unless you stop the service of course!
natecoyeAuthor Commented:
Unfortunately, Trend doesn't let you do this.  I get "the requested pause, continue, or stop is not valid for this service."  Even from services.msc, It won't let you stop the service.  You have to manually open trend software and input the password to unload the client.  It's cool that Sophos lets you do that, though.
I have this issue as well. Windows 2008R2, backup exec 2010 R3...Jobs are already starting to slow and will eventually get to 600mb/min.


Going to see if memory usage increases over the coming days to see if this effects speed as well. Never thought to check this.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.