Solved

Netbackup Master and media servers

Posted on 2013-06-07
7
2,033 Views
Last Modified: 2013-06-20
Hi Friends,

Need some suggestion on Netbackup. We are using it from many years however we are facing issues these days in exchange backups and VM backups. (Snapshot errors, error 130, error 156, partial backups).
We consulted Symantec and they are suggesting us to add media servers. Right now we have 3 Master servers in 3 regions and they are acting as a media server too.
Do we really need to add media servers. I am attaching an excel file with job count details. Do let me know if you experts need more details/information about our environment.
Experts-exchange.xlsx
0
Comment
Question by:TechJoshi
  • 4
  • 3
7 Comments
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 39230810
Backup environments tend to get neglected as IT environments grow, so backup infrastructure tends to bend, then break under the increased load as years go by. Symantec is almost certainly correct, and there's an easy way to check:

First you need to work out how many TB there is at each site. You also need to know your backup window - this mustn't exceed 12 hours. Once you have that, divide the number of TB by the backup window and multiply by 1000 to get the number of GB per hour (the throughput) you need to achieve to hit the backup window. Next step is to workout how many of the 'bottlenecks' you need to hit the throughput figure, so divide the throughput by:
504 GB/hr for LTO-5 tape drives
576 GB/hr for LTO-6 tape drives
You now have the number of tape drives going absolutely flat-out required to hit then backup window. Make sure you round up.
Compare this number to the number of tape drives you actually have. Now add 50% to that number as you'll not be able to drive tapes flat-out all the time (backup clients being what they are), so you need headroom.
For network links, divide the throughput figure by
180 GB/hr for 1Gb Ethernet
1800 GB/hr for 10Gb Ethernet
You now have the number of NICs and switch ports you need to hit the required  throughput.
If you exceed, say, four to eight 1Gb/sec NICs, then it's time to add a media server.

Symantec has a nifty solution with their NBU53x0 appliances - they give you de-duped backup to disk with a built-in media server and integrated tape-out, so they solve a bunch of problems at once
0
 

Author Comment

by:TechJoshi
ID: 39234555
Thanks meyersd! I will check these parameters and get back to you in case of any confusion.

-Rajat
0
 

Author Comment

by:TechJoshi
ID: 39240205
Attached is the data backed up in 12 hours window. Please advice is it moderate or high?
EE-12hoursData---Copy.xlsx
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 30

Accepted Solution

by:
Duncan Meyers earned 500 total points
ID: 39243523
That's very slow. You should be able to complete 537GB in a couple of hours, assuming LTO-5 or LTO-6 tapes. Even LTO-4 should complete within a couple of hours. Even if you were running your backups across a single 1Gb Ethernet link, they should finish with 3 or 4 hours.

There could be many causes of the poor performance:
- file servers with millions of small files are *always* slow
- Exchange brick-level backups are *always* slow. Use a database backup and Recovery Storage Groups instead.
- defrag Windows servers. File systems fragmentation can really hurt backups. Diskeeper is available here: http://www.condusiv.com/products/diskeeper/
- For Windows backups, try and group a number of clients into the same policy so you can get some concurrency happening
- read this: http://www.symantec.com/business/support/index?page=content&id=DOC4483
- try these tips:

Test read-speed from disk on media server to verify that data streams can be read as fast as the tape can write. Use the bpbkar test as per the Performance Tuning Guide:

Choose a folder/file system of at least 2GB in size.

1 Turn on the legacy bpbkar log by ensuring that the bpbkar directory exists.
UNIX
/usr/openv/netbackup/logs/bpbkar
Windows
install_path\NetBackup\logs\bpbkar
2 Set logging level to 1.
3 Enter the following:
UNIX
/usr/openv/netbackup/bin/bpbkar -nocont -dt 0 -nofileinfo -nokeepalives file_system > /dev/null
Where file_system is the path being backed up.
Windows
install_path\NetBackup\bin\bpbkar32 -nocont X:\ > NUL
Where X:\ is the path being backed up.
4 Check how long it took NetBackup to move the data from the client disk:
UNIX: The start time is the first PrintFile entry in the bpbkar log. The end time is the entry "Client completed sending data for backup." The amount of data is given in the entry "Total Size."
Windows: Check the bpbkar log for the entry "Elapsed time."

and

 1. Check client settings:
        - Confirm your data are not 1000's of small files if is the case than you need a different solution.
        - Confirm your data is traveling through the backup NIC and is configured 1gig/auto under the switch and OS levels
        - Review your NETWORK_BUFFER_SIZE or Buffer_Size for windows, highiest value can be 256KB
        - If you have logging enabled at verbose 5 can impact performance lower this to 0
        - Where is the data stored DAS, SAN or NAS and based on this you need to confirm you are not doing Cross Mount Points for NAS Qtrees, Check SAN side, how is the throughput, any errors at the switch level?, what RAID configuration you have?, normaly SAN is never the problem, but dont let anything open. DAS can be tested with iostat commands in unix and performance monitor in windows a bad drive can give you some pain.
        - Measure the Network Speed with simple FTP copy of a 10GB file to your media server this will tell you how the Network speed is doing, also you can open a ticket with Symantec to run SAS and detect Network problems.

    2. Media Server Settings:
        - Check NUMBER_DATA_BUFFERS and SIZE_DATA_BUFFERS, 128 and 256 are the average values, but ensure your shared memory is configured to support the:
NUMBER_OF_DRIVES * MULTIPLEXING * (NUMBER_DATA_BUFFERS * SIZE_DATA_BUFFERS) = TOTAL_MEMORY. If you dont have enough memory than you cannot set this values.
        - Same deal as the clients check NIC speeds settings and switches configs.
        - Check media server I/O using dd and mt commands to backup files to your drives and check each one, this will measure your throughput speed between media and drive.
        - Confirm your HBA's speed is at the highiest value and that is supported by your Fabric.
        - Search for errors at SAN switch level for your tape drives.
        - with iostat/kstat you can see the speed of all your drives and look for I/O and Transport Errors.

(from http://www.symantec.com/connect/forums/netbackup-65-how-troubleshoot-very-slow-backups)
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 39243526
The bottom line, though, is another media server, or a NBU5320 appliance is unlikely to help. If you're only getting 44GB per hour throughput, you have a problem in the backup environment that you need to fix first.
0
 

Author Comment

by:TechJoshi
ID: 39244520
Thanks! Very helpful.
What should we do for the servers with 1000's of files being backed up every week. I have seen that it is taking almost a week for thess clients and next full schedule is waiting for the same client.
0
 
LVL 30

Expert Comment

by:Duncan Meyers
ID: 39262484
Almost a week?? That's crazy. Can you open another question please and we should get to the  bottom of it. Please post a link to your new question here.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Having issues meeting security compliance criteria because of those pesky USB drives? Then I can help you! This article will explain how to disable USB Mass Storage devices in Windows Server 2008 R2.
How to fix error ""Failed to validate the vCentre certificate. Either install or verify the certificate by using the vSphere Data Protection Configuration utility" when you are trying to connect to VDP instance from Vcenter.
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now