Solved

Symantec Backup Exec

Posted on 2013-05-12
17
1,010 Views
Last Modified: 2016-11-23
We have two tape drive
HP StorageWorks 1/8 G2 LTO-4
Dell Power Vault Tl2000

Based on the attached screen dump. what's the current tape size that it can support right now ?

ON HP StorageWorks, the backup job rate is 2,162MB/min while the restore job rate is around 113MB/min (sometime it fails to even lower 50 MB/min). Do you have any idea ?
Is there a way to tune the restore job rate ?

Tks
Ultrium-Tape.bmp
0
Comment
Question by:AXISHK
  • 5
  • 5
  • 4
  • +1
17 Comments
 
LVL 10

Expert Comment

by:rjanowsky
ID: 39160388
The left one is a LTO Ultrium 4 and the right one a LTO Ultrium 3 (my LTO3 tapes have a capacity of 391 GB, but your's are quite close to that figure). But that are only the media statistics not the capabilities of the tape drive. The HP Storage Works is a LTO-4 drive and the the DELL Powervault could be anything between LTO-4, LTO-5 and LTO-6 - see the library specs at the Dell website.

The restore rate is depending on the type of the data (small versus big files, docs versus databases etc.) and is depends which workload exists at the backup and restore servers. Did you try running the restore jobs after normal work hours or at the weekend?
0
 

Author Comment

by:AXISHK
ID: 39160453
even within weekend, the restore rate run as 113MB/min. Is that normal ?
0
 
LVL 10

Expert Comment

by:rjanowsky
ID: 39160481
This seems to be very slow. Do you have an Antivirus solution installed at the servers? Have a look at this Symantec Backup Exec Tuning article.
0
 
LVL 20

Expert Comment

by:SelfGovern
ID: 39161830
Also note that HP has a free utility called HP Library and Tape Tools.  One of the things you can use it for is to simulate backup or restore jobs to help identify performance bottlenecks.

Download the utility from http://h18004.www1.hp.com/products/storageworks/ltt/index.html.
You can model a restore session in the tool.

If you're experiencing performance issues, I also recommend you make sure that your tape drives and libraries have the latest firmware.  L&TT can do this for your HP gear.

Can you describe or diagram your backup infrastructure?  That is, how are the tape drives connected to the servers?   What operating systems are involved?   What are you backing up (type of data, size of files, etc.)?   Is performance different when you run a job on drive A, versus drive B?
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39162775
Well restores are always slower than backups, but that does look a bit slow.

Mind you, your backups are not too speedy either 2.1GB/min is only 36MB/sec less than half what the drive is capable of, and only just above the minimum speed before it starts shoe-shining.

Maybe the minimum speed is why the restores are so slow?
0
 

Author Comment

by:AXISHK
ID: 39163435
NAV has been disabled but it doesnt' help.
2.1GB/min is less than half waht the drive is capable. Where can you tell this ? Is this the speed rate for LTO-4 tape ?


The Backup Exec media server is installed on Window 2008 R2 SP1 that is connected to HP StorageWorks 1/8 G2. The server will backup its local drives and few more servers through the agents.

We are building a tapes and it reads for 934Gb, with Job rate 133 MB/min. The Elasped Time is 111:34:42 and it hasnt' finished yet.

The tape statistic is
Used :590Gb
Available : 190Gb
Written : 900Gb
Read : 120Gb
Compression ration :  1.45:1

Any idea to try tunning the performance is appreciate ? In addiiton, how much data will it read as the tape only used 590Gb ? Tks
0
 
LVL 10

Assisted Solution

by:rjanowsky
rjanowsky earned 100 total points
ID: 39163639
2.1GB/min is less than half waht the drive is capable. Where can you tell this ? Is this the speed rate for LTO-4 tape ?
At this website is a chart describing the specs of the different LTO generations. Your LTO-4 can achieve uncompressed 120 MB/s or 7.2 GB/h. That are the maximums which - most often - won't be realized in a normal production environment with "normal" backup data.
Any idea to try tunning the performance is appreciate ? In addiiton, how much data will it read as the tape only used 590Gb ?
The 590 GB or compressed with a rate of 1.45:1 - so you have round about 855 GB uncompressed data on tape, which have to be processed.

I also recommend installing and using the HP LTT (see above) to diagnose your tape drive and get detailed info about the health of the drive.
0
 
LVL 20

Expert Comment

by:SelfGovern
ID: 39166098
LTO-4 has a native speed of 120MB/second -- 7,200 MB/minute, or over 400GB/hour.
If your data compresses at 1.45:1, it's full-speed writes will be at
174MB/sec, 10,440MB/minute, or 626GB/hour

LTO-4 tape cartridges are native 800GB.  You should be able to fit about  1.1TB of 1.45:1 compressible data on a single LTO-4 tape.

As slow as you are feeding the drive, I'd recommend you turn compression off -- you'll use more tape, but you'll stream more, and your tapes and tape drives will last much longer because they won't be continually stopping, rewinding and starting again as the buffer continually gets empty.

The first step in investigating performance issues of tape drives is to get on the latest rev of firmware for both the drive and the library/autoloader.  HP's free Library and Tape Tools (see post above) will do that for you simply.
0
Google Storage: Standard vs. Nearline vs. Coldline

Google Cloud Storage has a number of classes to choose from. Although there are a lot in common, they vary in price and usage terms. This post explains Google Cloud Storage classes and helps to understand which  one to choose.

 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39166183
Here is a summary of HP LTO drives

From HP QuickSpecs
          Cap	                        HP Speed     HP Data Rate Matching
LTO-1	100GB			        16MB/s	0-16MB/s
LTO-2	200GB			        24MB/s	0-24MB/s
LTO-2	200GB	Ultrium 448	HH	24MB/s	0-24MB/s
LTO-3	400GB	Ultrium 920	HH	60MB/s	20-60MB/s
LTO-3	400GB	Ultrium 960		80MB/s	27-80MB/s
LTO-4	800GB	Ultrium 1760	HH	80MB/s	33-80MB/s
LTO-4	800GB	Ultrium 1840		120MB/s	40-120MB/s
LTO-5	1500GB	Ultrium 3000	HH	140MB/s	47-140MB/s
LTO-5	1500GB	Ultrium 3280		140MB/s	47-140MB/s

LTO-6	3200GB			        210MB/s

Open in new window


Media durability 1,000,000 passes on any area of tape,
equates to over 20,000 end-to-end passes/260 full tape backups
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39166230
@rjanowsky - disagree that you cannot hit the tape speeds specified, if you feed the drive data fast enough it WILL do those speeds!

The problem is getting the data off disk fast enough to feed the drive not that the drives can't do it.

The figures in my previous post are for uncompressed data on tape, so with compression the drives will accept data even faster.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 39166280
We are building a tapes and it reads for 934Gb, with Job rate 133 MB/min. The Elasped Time is 111:34:42 and it hasnt' finished yet.

The tape statistic is
Used :590Gb
Available : 190Gb
Written : 900Gb
Read : 120Gb
Compression ration :  1.45:1

Any idea to try tunning the performance is appreciate ? In addiiton, how much data will it read as the tape only used 590Gb ? Tks


At 133MB/min, thats 2.2MB/sec it sounds like the drive is shoe-shining, ie you are not feeding the drive fast enough. The LTO-4 needs at least 33MB/s to hit streaming speed
0
 

Author Comment

by:AXISHK
ID: 39193814
Install the HP LTT and stop all the BackupExec service. Run the diagnosis but it tells that the device is busy.

However, I can't find any option about testing the tape device read/write speed. Does that provide this function ?

Actually, I still can't sort out the bottleneck (hdds or tape device) ?

Tks
0
 
LVL 20

Expert Comment

by:SelfGovern
ID: 39194422
Yes, you can test the tape read/write performance.   See the icon "Dev Perf" just two over from the big yellow help question mark (or from the menu, "Function" --> "Device Performance").  This will allow you to read from or write to the tape drive directly to/from memory.

If you want to test your hard drive read/write performance, use the "Sys Perf" icon, which lets you create a test data set customized to your needs, and then performs the indicated disk <--> memory reads or writes.

In my experience, it is rare that the tape drive is the bottleneck.  The Windows file system, and how backup applications have to access it, generates abhorrant performance.   The preponderance of  small and tiny files, the deeply nested directory structures, fragmentation, and long file/directory names all contribute.
0
 

Author Comment

by:AXISHK
ID: 39194604
Thanks. Does it mean that by comparing the result from "Sys Perf" to "Dev Perf", I need tell the bottleneck ?

Why it always say my device is busy ? I have stop all the Backup Exec service. Do I need to reboot the server ?

Based on the posting above,
LTO-4      800GB      Ultrium 1760      HH      80MB/s      33-80MB/s

My current backup rate is around 2,162 MB/min -> 36MB/s. Based on this result, the tape device should be fine for backup. Can we assume that the hardware is fine and should I focus on restoring, (Window, file system, SQL express come with BackupExec...)



Tks
0
 
LVL 20

Accepted Solution

by:
SelfGovern earned 300 total points
ID: 39196448
Looking at the sys perf results and the dev perf results may help you identify a bottleneck.  But maybe not.   For instance, in a backup server environment, the bottleneck could be the network  Or, it could be because you're using your server CPU to compress and/or encrypt the data -- CPU intensive operations that could certainly slow down backups.  It's also possible that your anti-virus program could be processing files upon access and slowing things down; this won't be identified by the L&TT tests (but they may give you more information and help you narrow things down).  Also -- if the data set you use for the Sys Test doesn't mirror what's on your disk (depth of directory tree and relative distribution of file sizes), it won't give you results that mean anything much.

You might have Removable Storage Management active, which can attach to tape devices and make them seem busy.   Check that.  If RSM is not active, then perhaps you can just remove the tape and library in Device Manager, and then rediscover it -- often better than restarting the server!

Your performance at 36MB/sec average is at the minimum for uncompressible data.  Since your data is actually 1.45:1 compressible, you're only writing 25MB/second to tape (that's how much data you have after the 36MB/sec gets compressed at 1.45:1).  If you can't get performance up, consider turning HW and SW compression off -- this will use a bit more tape, but it will ensure your tape drive and tapes last as long as possible.  (remember also that both compression and feed speed vary over time throughout the backup job -- you might have a string of 100-500MB files followed by a thousand performance-killing 500 byte to 10K-byte files.   Likewise, you could have hundreds of MB of 2:1 compressible data, followed by hundreds of MB of uncompressible files.   When you hit a stretch of small, highly compressible files, your write-to-tape speed may be dropping to 10MB/sec or less.  Turning off compression will at least keep compression from being a tape-killer.

And -- 36MB/second is not unusual for backup speed from Windows systems, unless you're running off of high performance enterprise class disk arrays, and backing up large files.  I think you'll find that your filesystem is your biggest issue.   You could verify this by having BE perform an image backup instead of a file backup.   In an image backup, you're backing up the disk starting at the first sector and reading sectors in order until you get to the last one.  No filesystem overhead == best possible performance.   We don't normally use image backups because a restore from an image backup will take much longer than a restore from a file-by-file backup.

What exactly are you trying to restore that's giving you those 50 - 115MB/sec restores?  Is it the whole tape, or just some file data?   How big is the restore set in GB?

If you're just restoring a small subset, and you choose (as many would) a directory or two, and that directory happens to contain a bunch of small files and has a deep directory structure, you will see very slow performance -- the restore has to create the directory tree and allocate space for each file one at a time, which is a very time-consuming process.  Your antivirus could be contributing to the slowdown, too, if it is checking the files as they are written; consider turning it off for your test restores.   While you're doing a restore, monitor CPU performance in Windows to see if anything unexpected is happening.  

Try restoring large files only, and see if that gives you higher performance.

I know of one gent who found that restoring to a freshly-formatted drive was significantly faster than restoring to a drive that had been in use for a while; keeping your disk defragmented may improve your restore (and backup) speeds.
0
 

Author Comment

by:AXISHK
ID: 39198875
It can't recognize the tape driver. Any idea ?
HP-Tape-drive.bmp
0
 
LVL 16

Assisted Solution

by:Gerald Connolly
Gerald Connolly earned 100 total points
ID: 39199179
... Your performance at 36MB/sec average is at the minimum for uncompressible data.  Since your data is actually 1.45:1 compressible, you're only writing 25MB/second to tape ...

Looking another way, with a minimum speed of 33MB/s you need to feed the drive at 33*1.45MB/s, or approx 48MB/s Minimum
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

A quick step-by-step overview of installing and configuring Carbonite Server Backup.
This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…
This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now