BackupExec Performance Issues - Ideas?
I have been with Veritas tech support multiple times on this and gotten nowhere. We have BackupExec 15, version 14.2 and for the past year have had this problem- backups run slow... after they previously ran fast. The best way to illustrate this is as follows:
We have a hyper-v cluster. The backup server is on the same node and network as the fileserver in this case. The backup server is virtual, and the fileserver is virtual (I know that's redundant- just being clear). We have been going through these cycles where we rebuild the BE server and performance is great, then it tapers off and goes to tolerable, then abysmal. So we have two hyper-v nodes on a shared SAN array, and both machines are on the same node and LAN- that backs up to a Drobo 810 over iSCSI.
Two months ago, we backed up our fileserver, average data rate was 3500 MB/min. With no changes made, we ran the backup the following month and got 132 MB/min. I spent the month trying to get faster and couldn't. I built a new BE server and was back up to 3500 MB/Min. Veritas support has had us run their diagnostic tools and they keep saying the problem is a 1% drop across the network connection, and they keep saying the issue is the firewall, and they cannot point us to a problem on the server. Part of the problem with their diagnosis is that we turn off the Windows Firewall, and the network literally does not leave the hardware between the backup source and target servers, and then only leaves the VM to get to the storage.... plus the fact that rebuilding the server both on the same node, and on a completely separate hypervisor restores the performance. We are not backing up to tape, everything is to hard drives, and I am out of ideas. We have found that having multiple backups running simultaneously ridiculously exacerbates the problem- it will take a 3500 MB/Minute backup, then the second backup will run at 300MB/Min while the first one drops to 1200MB/minute, even on a fresh rebuild. To me this all points to a problem with the database or some issue within BE. Has anyone had a similar experience, and is so, how did you resolve it?
Thanks!
We have a hyper-v cluster. The backup server is on the same node and network as the fileserver in this case. The backup server is virtual, and the fileserver is virtual (I know that's redundant- just being clear). We have been going through these cycles where we rebuild the BE server and performance is great, then it tapers off and goes to tolerable, then abysmal. So we have two hyper-v nodes on a shared SAN array, and both machines are on the same node and LAN- that backs up to a Drobo 810 over iSCSI.
Two months ago, we backed up our fileserver, average data rate was 3500 MB/min. With no changes made, we ran the backup the following month and got 132 MB/min. I spent the month trying to get faster and couldn't. I built a new BE server and was back up to 3500 MB/Min. Veritas support has had us run their diagnostic tools and they keep saying the problem is a 1% drop across the network connection, and they keep saying the issue is the firewall, and they cannot point us to a problem on the server. Part of the problem with their diagnosis is that we turn off the Windows Firewall, and the network literally does not leave the hardware between the backup source and target servers, and then only leaves the VM to get to the storage.... plus the fact that rebuilding the server both on the same node, and on a completely separate hypervisor restores the performance. We are not backing up to tape, everything is to hard drives, and I am out of ideas. We have found that having multiple backups running simultaneously ridiculously exacerbates the problem- it will take a 3500 MB/Minute backup, then the second backup will run at 300MB/Min while the first one drops to 1200MB/minute, even on a fresh rebuild. To me this all points to a problem with the database or some issue within BE. Has anyone had a similar experience, and is so, how did you resolve it?
Thanks!
ASKER
The problem is that this keeps reoccurring. If I rebuild the backup server, backup times go right back to 3500, but if I go back to the previousVM that ran the backups, they are slow again. If we were running into the problem in consistent, consecutive backups, I would look at the environment, but the issue seems to follow the server.... and we are in another iteration of this cycle of rebuilding the server.
I don't quite understand
So you have now 2 BE servers running ?
if I go back to the previousVM that ran the backups, they are slow again
So you have now 2 BE servers running ?
ASKER
Not simultaneously. I never shut the previous one off, but both use the Drobo for storage, so my process is to mount the Drobo on the more recent VM, run the backups from there. If there are complicated backups, I wait until the backups on the newer VM run, and once complete, I shut down and hold all jobs on that server, and unmount the Drobo. Then I go to the previous server, mount the Drobo there, then start any backups that I need to run from there. The main reason for all of this is that the previous month's backups did not successfully run , so I am manually running from the faster backup the few large ones that failed, and switching to the slower one to run the bulk tiny backups. This is a very temporary process, just to make sure I get a full backup set this month, and I'm hoping to fix the problem with the previous BE server. By the way I forgot to include some important details- I intentionally am running 2012 servers -not R2- for both the hypervisors and the guest OS on the backup servers- also so I can trace back and know consistency between the environments as I troubleshoot.
OK... to be clear
You have 2 VM, both on same node and both are using same NIC (virtual switch) - I assume you're using Hyper-V (please correct me if I'm wrong)
On First server you can make backup with SUCCESS but ist's very slow
On second, new BE installation you can run backup FAST but there is a few errors so your backup status is not complete.
Are you using same login credentials on both BE ?
Maybe it would be faster repair problem with backup on second BE since spead issue is very hard to investigate/
You have 2 VM, both on same node and both are using same NIC (virtual switch) - I assume you're using Hyper-V (please correct me if I'm wrong)
On First server you can make backup with SUCCESS but ist's very slow
On second, new BE installation you can run backup FAST but there is a few errors so your backup status is not complete.
Are you using same login credentials on both BE ?
Maybe it would be faster repair problem with backup on second BE since spead issue is very hard to investigate/
ASKER
I'll try to make this a little clearer! I Have two physical environments. Both are on the same LAN. One is a 2-node Hyper-V with a shared SAN that contains the original backup server and the file server, both on the same node and LAN. These back up to a Drobo.
The second environment is a hypervisor that contains a single VM that is this second BE server. The only reason it exists is that I wanted to create a test environment that I could use to rule out elements of the problem- for example, if the backup is slow on the original environment, but fast on this test environment, the we know the problem is not the connection to the source data or the target storage- because in that instance, speed would be slow for either backup, not just one. All login credentials are the same. WE have been rebuilding backup servers every two months, and that solves the problem, but we simply don't have the resources to constantly rebuild servers to keep the backup going, and this problem appears to be something that is software-related.
The second environment is a hypervisor that contains a single VM that is this second BE server. The only reason it exists is that I wanted to create a test environment that I could use to rule out elements of the problem- for example, if the backup is slow on the original environment, but fast on this test environment, the we know the problem is not the connection to the source data or the target storage- because in that instance, speed would be slow for either backup, not just one. All login credentials are the same. WE have been rebuilding backup servers every two months, and that solves the problem, but we simply don't have the resources to constantly rebuild servers to keep the backup going, and this problem appears to be something that is software-related.
multiple things involves in performance issue . those are minor issues but different to find out what exactly causing .
make sure backup application is up to date and you need to perform assembly work like unplug and re plug the SCSI cables which is connected and make sure language cable which is connected to back up server is working fine .
use perform tool a built in tool from windows 2008 server you can monitor and get the output of server performance and know about what is causing the performance and final option is reboot the windows server
make sure backup application is up to date and you need to perform assembly work like unplug and re plug the SCSI cables which is connected and make sure language cable which is connected to back up server is working fine .
use perform tool a built in tool from windows 2008 server you can monitor and get the output of server performance and know about what is causing the performance and final option is reboot the windows server
Are you sure that after rebuild BE on second VM you're not installing any WIndows or BE updates after first backup ?
You've said first backup after server rebuild is fast and then after 2 month it's acting very fast so you must rebuild server again to get backup running 3500MBps
My question is:
Any Windows, SQL, or BE updates between ?
You've said first backup after server rebuild is fast and then after 2 month it's acting very fast so you must rebuild server again to get backup running 3500MBps
My question is:
Any Windows, SQL, or BE updates between ?
ASKER
None of those work- all cables have been plugged in for months, and the performance has changed based on the VM. we happen to encounter performance issues one month after great performance the prior month. Servers get rebooted every 2-3 weeks, especially the backup servers, which get rebooted every few days. The backup servers are dedicated servers- they do nothing except run BE. No SCSI cables are involved- it's all network cables.
ASKER
BE was updated most recently to troubleshoot the issue AFTER it occurred this time. Windows updates happen the last or first weekend of the month- but I don't see that being the issue because building a fresh server with the same Windows updates on it does not reproduce the problem. We have not made any SQL changes, so if any SQL updates, then they would be part of Windows Updates.
then I suggest that deploy a new server and do not install any patch's and Antivirus and install. the. backup server application and try . you will come to know the some conclusion
ASKER
We have done that 6 times in the past year. No change. we just have to rebuild every couple of months.
may be chances are with network droppings in the environment
ASKER
No network drops- it works fast some months then the next month, it runs at 1/10 or 1/20 the speed of the prior month.
It looks like by the time you database is growing (because of Job history, logs and catalogs) server performance is going down.
If you want to confirm if this is database related issue, make a database backup when server is slow, recreate VM and BE and restore DB.
If first backup will work slow then bingo.
If you have other SQL in your company maybe it would be better to install Symantec BE with connection to this SQL instead of SQL Express on VM.
That's my thoughts.
If you want to confirm if this is database related issue, make a database backup when server is slow, recreate VM and BE and restore DB.
If first backup will work slow then bingo.
If you have other SQL in your company maybe it would be better to install Symantec BE with connection to this SQL instead of SQL Express on VM.
That's my thoughts.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Best solution provided. No more other questions from author
Maybe the issue you have is in NIC buffer ?