Our Backups take a long time!

Hi,

Here are the stats:
-----------------------------------------------------------
For Exchange disk to disk backup:
-----------------------------------------------------------
Job name            : Exchange Server Backup - Disk to Disk-LegacyJob type            : BackupJob status          : SuccessfulJob log             : C:\Program Files\Symantec\Backup Exec\Data\BEX_TRMVC1_01378.xmlServer name         : TRMVC1Selection list name : Aleks- Symantec - Legacy - IS
Device name         : ExchangeTarget name         : ExchangeMedia set name      : TRMDISKDAILY
All Media UsedB2D000224
 

Job Summary Information
Byte count          : 50,870,785,493 bytesJob rate            : 274.60 MB/Min (Byte count of all backup sets divided by Elapsed time for all backup sets)
Files               : 0Directories         : 0Skipped files       : 0Corrupt files       : 0Files in use        : 0
Original start time : Friday, 27 June 2008 11:00:00 PMJob started         : Monday, 30 June 2008 11:00:05 PMJob ended           : Tuesday, 1 July 2008 4:00:44 AMElapsed time        : 05:00:39
-----------------------------------------------------------

-----------------------------------------------------------
For Network based Disk to Disk of all other Data
-----------------------------------------------------------

Job name            : Disk To Disk MondayJob type            : BackupJob status          : SuccessfulJob log             : C:\Program Files\Symantec\Backup Exec\Data\BEX_TRMVC1_01377.xmlServer name         : TRMVC1Selection list name : Backup Disk To Disk Monday
Device name         : TRMD2DTarget name         : TRMD2DMedia set name      : TRMDISKDAILY
All Media UsedB2D000371B2D000367B2D000418B2D000411B2D000457B2D000385B2D000454B2D000458B2D000444B2D000460B2D000456B2D000409B2D000462B2D000399B2D000370B2D000465B2D000415B2D000374B2D000406B2D000401B2D000413B2D000466B2D000449B2D000463B2D000448B2D000373B2D000451B2D000441B2D000414B2D000437B2D000379B2D000412B2D000443B2D000436B2D000445B2D000416B2D000369B2D000407B2D000450B2D000402B2D000438B2D000405B2D000439B2D000372B2D000452B2D000442B2D000459B2D000453B2D000455B2D000403B2D000408B2D000410B2D000447B2D000428B2D000446
 

Job Summary Information
Byte count          : 584,961,339,536 bytesJob rate            : 1,283.82 MB/Min (Byte count of all backup sets divided by Elapsed time for all backup sets)
Files               : 778,868Directories         : 140,774Skipped files       : 0Corrupt files       : 0Files in use        : 0
Original start time : Monday, 23 June 2008 9:00:00 PMJob started         : Monday, 30 June 2008 9:00:04 PMJob ended           : Tuesday, 1 July 2008 9:38:21 AMElapsed time        : 12:38:17
 
-----------------------------------------------------------
Now For Duplicate/copy from Disk to tape
-----------------------------------------------------------
-----------------
First copy of exchange:
------------------

Set type               : DuplicateSet status             : CompletedSet description        :
Resource name          : \\TRMEXCH1\System?StateLogon account          : Encryption key         : No
 


Byte count             : 1,244,711,684 bytesRate                   : 757.00 MB/Min
Files                  : 0Directories            : 1Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Tuesday, 1 July 2008 9:39:22 AMEnd time               : Tuesday, 1 July 2008 9:40:56 AM
Media used             : 000001L3

--------------------------------
sample copy of othe data
---------------------------------
Set type               : DuplicateSet status             : CompletedSet description        :
Resource name          : \\TRMVC1\J:Logon account          : Encryption key         : No
 


Byte count             : 17,150,133,072 bytesRate                   : 1,395.00 MB/Min
Files                  : 576Directories            : 3Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Tuesday, 1 July 2008 9:50:36 AMEnd time               : Tuesday, 1 July 2008 10:02:21 AM
Media used             : 000001L3


----------------
Then Verify
-------------------------
Set type               : VerifySet status             : CompletedSet description        : Disk To Disk Monday
Resource name          : \\TRMFILE1\F:Logon account          :

Byte count             : 164,930,373,587 bytesRate                   : 2,222.00 MB/Min
Files                  : 305,251Directories            : 12,527Skipped files          : 0Corrupt files          : 0Files in use           : 0
Start time             : Tuesday, 1 July 2008 6:30:31 PMEnd time               : Tuesday, 1 July 2008 7:41:17 PM

-----------------------------------------------------------------------------------------------------

I hope this makes sense, I've tried to copy out of the logs as much definitive info as possible.

cheers...

Hi,

Well to answer your question, here is the setup:
the 2 VMWARE ESX Servers (connected to SAN with dual 4 GB HBAs) have all the windows 2003 servers stored on the SAN that are being backed up, except the physical BackupExec DELL 1950 server which also has HBA connected to the SAN for the temporary DISK TO DISK STORE about 1.2 TB or so of space.  They are all on the same SAN.

My reservation has always been that the DATA and servers being backed up have to travel out of the SAN via the ESX servers through the network to another 2003 server and then saved to SAN again, then copied to TAPE.

in regards to the 8 hour GAP you mentioned, actually that's not the case, since (4) I pasted into the message is only part of the full backup set, and I only put it here as an example of the total process.  (5) correctly only begins when all of (4) ends.  Which is roughly the 580GB of data.  That part actually works correctly.  But regardless, the time windows is really bad.  my SAN has a 4GB cache buffer and the Servers themselves are on 146Gb 10k RAID 10 firbre arrays,  the majority of DATA is on 300GB 10k RAID 5 fibre arrays.  The exchange runs on a combination of these two.  Admittedly they are a 2GB loop and not the 4GB loop that the HBAs can depend on.  But seriously I'm running a SAN intended for an environment of thousands of users with an environment of 40 or so.

Can you tell me what sort of speeds you would expect to get with normal direct attached SCSI RAID arrays and copying via a normal Gigabit network (my ESX have 4 gigabit NICs teamed and should definitely put out more through-put than the average servers people run, and my switches have the switching capacity for it).

I'd appreciate any help in this regard...  also if anyone knows about running the two Dell and HP tape arrays together...

Thanks...

So what does this mean?

"My reservation has always been that the DATA and servers being backed up have to travel out of the SAN via the ESX servers through the network to another 2003 server and then saved to SAN again, then copied to TAPE"

That you move the data across your IP network?    If its not please detail the data-flow.

A SCSI attached RAIDarray with all things being equal (Number of disks and equivalent speeds) should perform similar to your existing SAN.

But if you really are moving the data across your network remember that NIC teaming gives you redundant connections but each connection is still only gigabit speed (ie max 100MB/sec) and very few applications move data very efficiently across ethernet anyway so transfer rates are typically bad and of course that is what you are seeing.

Lose the copy!

It will probably be quicker to lose the disk-2-disk backup all together and backup direct to tape.

G

Just worked it out!

    580GB + 50GB = 630GB

    LTO-3 uncompressed = 80MB/sec

    630GB @80MB/sec = <3 Hours

G

Wow, that much?

okay the numbers you have raised are going to generate a lot of questioning from my hardware and software configurators today.

In terms of the "So what does this mean?"My reservation has always been that the DATA and servers being backed up have to travel out of the SAN via the ESX servers through the network to another 2003 server and then saved to SAN again, then copied to TAPE""

What I meant is that the whole backup solution is not what I was originally anticipating of having.  Running a fibre SAN, I was originally hoping to directly backup via fibre, but the costs were prohibitive.  The other issue is that the main AD/Exchange/file/print and other servers are all virtualised VMs and I'm only backing up from inside the VMs, so for a disaster recovery, I need to setup the ESX and base windows VMs with the correct LUNs/drives attached, install backup exec agent and then restore from the backup.  The whole process seems very inefficient to me.  That was my meaning, however I don't see a quick and cheap way of making a leap in efficiency, hence I'm hoping to stick with current plans and improve speed of backups and only reconfigure to two tapes/backup as I'm running out of space to backup my data on a single tape.  Hence I bought the extra Tape library.  Do you have any thoughts on wether it is possible to integrate the new HP MSL6030 into the current DELL configured system?

As far as backup speeds and limitations go.  If you imagine that you can normally get roughly 70% throughput out of a gigabit link, therefore you get say 700Mbs => 700/8 => 87.5 MB/s is what you may get on a good working LAN connection (am I right?).  Hence the backup speeds you just quoted for tape are equivalent to good LAN speeds anyway, even if your LAN backup speed was 50% degraded, I should still be able to backup up Disk to Disk in half the time it is achieving now.

As far as Disk to Tape, that is another worry, why I can't manage to backup through the HBA (mind you the SAN LUN is RAID 5 on 500GB SATA arrays for the disk-to-disk temporary store) is really perplexing, as I should be getting closer to the 3 hours and not the 6 hours I get now.(currently getting 12 hours including the full verify)

Your comments are very much appreciated.  Especially the realistic speeds you have mentioned.

Thank you.

Have you thought about using snapshots on the RAIDarray and then backing those up to tape?

hmm,

Do you mean using backup exec or some other thirdparty snapshoting software?
Or do you mean using some EMC SAN snapshotting process to snap shot into a different LUN and using backup exec or other software to backup to tape?

Or did I misunderstand the whole thing?

Well i was thinking about snapshots on the RAIDarray itself!
That way you can assign the snapshot to your backup server mount it there and backup direct from the SAN.

Of course if you don't have the appropriate licence i expect EMC charge a ton for it!   :-)

Hi,

sorry to respond so late connollyq!

I imagine you're right about the EMC charge for such a service.  To be frank, the SAN was purchased at an unbelievably cheap price and as such lacks the added software that other end users normally purchase with a SAN like this.  That's the only reason a small SMB could afford to buy it.

Nevertheless, the problem still exists and I feel that general performance on the network and more so illustrated on the backup service tells me that something is definitely wrong here, in that either the SAN doesn't perform to spec or there is a huge problem in how the server's are configured in using the SAN.

At the same time however, I'm still in a situtation where I need to expand my backup tape ability, hence the purchase of teh 30x1 LTO 3 tape library to go into service in addition to my existing 8x1 LTO 3 library.  My question regarding this is wether I can daisy chain the new library wich is HP with the old library which is DELL(IBM) and still connected all to the same backup exec server and reconfigure backup jobs to either share the load or do 2tape backups (daily's going to the 30x1 and weekly going to the 8x1, or something like that).  I'm not a backup expert so I need to do some research on this.

First question is, can I daisy chain these and connect to my exising backup exec server via SCSI?

thanks..

I believe my recommendations are valid