Solved

RSync & NTBackup dilemma - compression

Posted on 2006-10-30
7
1,755 Views
Last Modified: 2013-12-01
The question: How can I make the most of rsync running under Windows and get it to store files in a compressed format?

The details:  I'm using rsync (DeltaCopy) and NTBackup to remotely backup one Windows 2003 server to another across broadband links. I'm looking for help from EE to try and determine the best method using the two apps or preferably just one.  Right now I'm doing nightly differential backups with NTBackup which includes the System State.  The resulting file is approx. 2GB and slowly growing.  It has about doubled in size in a month and I'm guessing it will continue to grow unless I do another full backup.  

I'm using NTBackup for the following reasons:
1 It gets the System State
2 It uses Shadow Copy to get all open files
3 It saves in one neat and easy file
4 COMPRESSION

Using rsync to get my .bkf file to the remote server sped things up significantly over XCopy.  It still says it's copying 2GB but I think it's telling me the entire size of the file it's copying, even if it only copies some of the bits.  It takes just under an hour to copy the 2GB over DSL/Cable lines.  

My biggest problem with using rsync alone is that I would miss out on the storage compression.  I know rsync compresses data to transfer over the wire but does it store the files on the remote end compressed?  I need to have it compressed on the remote server to save storage space.  I know about hardware compressing the drive, but isn't that additional compression beyond the software compression?  What I mean is, if I have a 2GB file and compress it to 1GB with NTBackup and then store it on a compressed drive, won't it still compress further?   I think with rsync, it may compress the 2GB file for transfer but I end up with a 2GB file on the other side (or ~1GB) on a compressed drive.  Is that correct?

Thanks in advance.
0
Comment
Question by:VortexAdmin
  • 2
  • 2
7 Comments
 
LVL 87

Accepted Solution

by:
rindi earned 250 total points
ID: 17848449
No, rsync just synchronizes the source with the destination. It won't compress the file on the destination. Anyway, isn't the bkf file you have created with ntbackup already compressed? A compressed file is usually not further compressable, or only to a very small extent which isn't worth the time. If the bkf file isn't compressed, then look at a commercial backup product for your server like veritas backupexec or arcserve which do compress the backup files.
0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 17851942
The .bkf is compressed, but I was trying to figure out a way not to have to use NTBackup at all. There would be some benefits to having the full copy on the remote server, instead of in a single .bkf file.  The reason I'm still using NTBackup though is because A) the files aren't compressed once they're on the remote server using just rsync and B) I need to be able to get the System State.

It looks to me like when a .bkf file is created on the source, and it includes the System State, then rsynced with the previous day's .bkf file on the remote server, that more than just the changed bits are being copied over.  I say this because I use a VPN connection between the two and it tells me 2+ GB has passed through the connection and that's about the size of my .bkf file.  However, it does seem to happen a lot faster than simple XCopy so maybe I need to do some experimenting and see what's going on for sure, ie. whether I'm actually benefiting from rsync delta copies or if rsync is just inherently faster at copying than XCopy without delta being a factor.  Do you use rsync?  Do people always combine it with some other method (such as NTBackup) to make it effective for backup?

Thanks.
0
 
LVL 87

Expert Comment

by:rindi
ID: 17856093
I haven't used rsync yet, but it is on my list for testing when I can get the chance, as it is supposed to be perfect. I never use xcopy though, rather robocopy, which is a free commandline utility in the windows 2000 server resource kit tools. To get the system state you do need a dedicated software like ntbackup.
0
 
LVL 8

Assisted Solution

by:Disorganise
Disorganise earned 250 total points
ID: 17871509
I imagine the reason rsync appears faster is because it's been optimised for WAN communication, whereas Xcopy is very locally oriented.  ie, the number of confirmation and ordering packets created by xcopy would be significantly larger than the number generated by rsync I would imagine.  it's the same argument for why WAFS speeds up WAN file performance.

0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 18074600
Sorry about that, didn't close the question because I never got a good answer.  But I appreciate the responses and agree with the recommendations for a split. Thanks.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

AWS Glacier is Amazons cheapest storage option and is their answer to a ‘Cold’ storage service.  Customers primarily use this service for archival purposes and storage of infrastructure backups.  Its unlimited storage potential and low storage cost …
I previously wrote an article addressing the use of UBCD4WIN and SARDU. All are great, but I have always been an advocate of SARDU. Recently it was suggested that I go back and take a look at Easy2Boot in comparison.
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…
This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now