BackupExec Performance Issues - Ideas?

I have been with Veritas tech support multiple times on this and gotten nowhere. We have BackupExec 15, version 14.2 and for the past year have had this problem- backups run slow... after they previously ran fast. The best way to illustrate this is as follows:
We have a hyper-v cluster. The backup server is on the same node and network as the fileserver in this case. The backup server is virtual, and the fileserver is virtual (I know that's redundant- just being clear). We have been going through these cycles where we rebuild the BE server and performance is great, then it tapers off and goes to tolerable, then abysmal. So we have two hyper-v nodes on a shared SAN array, and both machines are on the same node and LAN- that backs up to a Drobo 810 over iSCSI.

Two months ago, we backed up our fileserver, average data rate was 3500 MB/min. With no changes made, we ran the backup the following month and got 132 MB/min. I spent the month trying to get faster and couldn't. I built a new BE server and was back up to 3500 MB/Min. Veritas support has had us run their diagnostic tools and they keep saying the problem is a 1% drop across the network connection, and they keep saying the issue is the firewall, and they cannot point us to a problem on the server. Part of the problem with their diagnosis is that we turn off the Windows Firewall, and the network literally does not leave the hardware between the backup source and target servers, and then only leaves the VM to get to the storage.... plus the fact that rebuilding the server both on the same node, and on a completely separate hypervisor restores the performance. We are not backing up to tape, everything is to hard drives, and I am out of ideas. We have found that having multiple backups running simultaneously ridiculously exacerbates the problem- it will take a 3500 MB/Minute backup, then the second backup will run at 300MB/Min while the first one drops to 1200MB/minute, even on a fresh rebuild. To me this all points to a problem with the database or some issue within BE. Has anyone had a similar experience, and is so, how did you resolve it?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Tom CieslikIT EngineerCommented:
Did you try copy a LARGE amount of data over network from/to backup server and from/to VM ?
Maybe the issue you have is in NIC buffer ?
it_medcompAuthor Commented:
The problem is that this keeps reoccurring. If I rebuild the backup server, backup times go right back to 3500, but if I go back to the previousVM that ran the backups, they are slow again. If we were running into the problem in consistent, consecutive backups, I would look at the environment, but the issue seems to follow the server.... and we are in another iteration of this cycle of rebuilding the server.
Tom CieslikIT EngineerCommented:
I don't quite understand
if I go back to the previousVM that ran the backups, they are slow again

So you have now 2 BE servers running ?
CEOs need to know what they should worry about

Nearly every week during the past few years has featured a headline about the latest data breach, malware attack, ransomware demand, or unrecoverable corporate data loss. Those stories are frequently followed by news that the CEOs at those companies were forced to resign.

it_medcompAuthor Commented:
Not simultaneously. I never shut the previous one off, but both use the Drobo for storage, so my process is to mount the Drobo on the more recent VM, run the backups from there. If there are complicated backups, I wait until the backups on the newer VM run, and once complete, I shut down and hold all jobs on that server, and unmount the Drobo. Then I go to the previous server, mount the Drobo there, then start any backups that I need to run from there. The main reason for all of this is that the previous month's backups did not successfully run , so I am manually running from the faster backup the few large ones that failed, and switching to the slower one to run the bulk tiny backups. This is a very temporary process, just to make sure I get a full backup set this month, and I'm hoping to fix the problem with the previous BE server. By the way I forgot to include some important details- I intentionally am running 2012 servers -not R2- for both the hypervisors and the guest OS on the backup servers- also so I can trace back and know consistency between the environments as I troubleshoot.
Tom CieslikIT EngineerCommented:
OK... to be clear
You have 2 VM, both on same node and both are using same NIC (virtual switch) - I assume you're using Hyper-V (please correct me if I'm wrong)

On First server you can make backup with SUCCESS but ist's very slow
On second, new BE installation you can run backup FAST but there is a few errors so your backup status is not complete.

Are you using same login credentials on both BE ?
Maybe it would be faster repair problem with backup on second BE since spead issue is very hard to investigate/
it_medcompAuthor Commented:
I'll try to make this a little clearer! I Have two physical environments. Both are on the same LAN. One is a 2-node Hyper-V with a shared SAN that contains the original backup server and the file server, both on the same node and LAN. These back up to a Drobo.
The second environment is a hypervisor that contains a single VM that is this second BE server. The only reason it exists is that I wanted to create a test environment that I could use to rule out elements of the problem- for example, if the backup is slow on the original environment, but fast on this test environment, the we know the problem is not the connection to the source data or the target storage- because in that instance, speed would be slow for either backup, not just one. All login credentials are the same. WE have been rebuilding backup servers every two months, and that solves the problem, but we simply don't have the resources to constantly rebuild servers to keep the backup going, and this problem appears to be something that is software-related.
SAM ITAD windows Admin  Commented:
multiple things involves in performance issue . those are minor issues but different to find out what exactly causing .

make sure backup application is up to date and you need to perform assembly work like unplug and re plug the SCSI cables which is connected and make sure language cable which is connected to back up server is working fine .
use perform tool a built in tool from windows 2008 server you can monitor and get the output of server performance and know about what is causing the performance and final option is reboot the windows server
Tom CieslikIT EngineerCommented:
Are you sure that after rebuild BE on second VM you're not installing any WIndows or BE updates after first backup ?
You've said first backup after server rebuild is fast and then after 2 month it's acting very fast so you must rebuild server again to get backup running 3500MBps

My question is:
Any Windows, SQL, or BE updates between ?
it_medcompAuthor Commented:
None of those work- all cables have been plugged in for months, and the performance has changed based on the VM. we happen to encounter performance issues one month after great performance the prior month. Servers get rebooted every 2-3 weeks, especially the backup servers, which get rebooted every few days. The backup servers are dedicated servers- they do nothing except run BE. No SCSI cables are involved- it's all network cables.
it_medcompAuthor Commented:
BE was updated most recently to troubleshoot the issue AFTER it occurred this time. Windows updates happen the last or first weekend of the month- but I don't see that being the issue because building a fresh server with the same Windows updates on it does not reproduce the problem. We have not made any SQL changes, so if any SQL updates, then they would be part of Windows Updates.
SAM ITAD windows Admin  Commented:
then I suggest that  deploy a new server and do not install any patch's and Antivirus and install. the. backup server application and try . you will come to know the some conclusion
it_medcompAuthor Commented:
We have done that 6 times in the past year. No change. we just have to rebuild every couple of months.
SAM ITAD windows Admin  Commented:
may be chances are with network droppings in the environment
it_medcompAuthor Commented:
No network drops- it works fast some months then the next month, it runs at 1/10 or 1/20 the speed of the prior month.
Tom CieslikIT EngineerCommented:
It looks like by the time you database is growing (because of Job history, logs and catalogs) server performance is going down.

If you want to confirm if this is database related issue, make a database backup when server is slow, recreate VM and BE and restore DB.
If first backup will work slow then bingo.

If you have other SQL in your company maybe it would be better to install Symantec BE with connection to this SQL instead of SQL Express on VM.

That's my thoughts.
Tom CieslikIT EngineerCommented:
Another thoughts is set BE to keep catalogs and job history only let say 30 days and observer server ?

You can try truncate in Original VM BE whole DB and logs and run backup. If SQL is an issue backup should start working much faster.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Tom CieslikIT EngineerCommented:
Best solution provided. No more other questions from author
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.