Solved

RSync & NTBackup dilemma - compression

Posted on 2006-10-30
7
1,757 Views
Last Modified: 2013-12-01
The question: How can I make the most of rsync running under Windows and get it to store files in a compressed format?

The details:  I'm using rsync (DeltaCopy) and NTBackup to remotely backup one Windows 2003 server to another across broadband links. I'm looking for help from EE to try and determine the best method using the two apps or preferably just one.  Right now I'm doing nightly differential backups with NTBackup which includes the System State.  The resulting file is approx. 2GB and slowly growing.  It has about doubled in size in a month and I'm guessing it will continue to grow unless I do another full backup.  

I'm using NTBackup for the following reasons:
1 It gets the System State
2 It uses Shadow Copy to get all open files
3 It saves in one neat and easy file
4 COMPRESSION

Using rsync to get my .bkf file to the remote server sped things up significantly over XCopy.  It still says it's copying 2GB but I think it's telling me the entire size of the file it's copying, even if it only copies some of the bits.  It takes just under an hour to copy the 2GB over DSL/Cable lines.  

My biggest problem with using rsync alone is that I would miss out on the storage compression.  I know rsync compresses data to transfer over the wire but does it store the files on the remote end compressed?  I need to have it compressed on the remote server to save storage space.  I know about hardware compressing the drive, but isn't that additional compression beyond the software compression?  What I mean is, if I have a 2GB file and compress it to 1GB with NTBackup and then store it on a compressed drive, won't it still compress further?   I think with rsync, it may compress the 2GB file for transfer but I end up with a 2GB file on the other side (or ~1GB) on a compressed drive.  Is that correct?

Thanks in advance.
0
Comment
Question by:VortexAdmin
  • 2
  • 2
7 Comments
 
LVL 88

Accepted Solution

by:
rindi earned 250 total points
ID: 17848449
No, rsync just synchronizes the source with the destination. It won't compress the file on the destination. Anyway, isn't the bkf file you have created with ntbackup already compressed? A compressed file is usually not further compressable, or only to a very small extent which isn't worth the time. If the bkf file isn't compressed, then look at a commercial backup product for your server like veritas backupexec or arcserve which do compress the backup files.
0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 17851942
The .bkf is compressed, but I was trying to figure out a way not to have to use NTBackup at all. There would be some benefits to having the full copy on the remote server, instead of in a single .bkf file.  The reason I'm still using NTBackup though is because A) the files aren't compressed once they're on the remote server using just rsync and B) I need to be able to get the System State.

It looks to me like when a .bkf file is created on the source, and it includes the System State, then rsynced with the previous day's .bkf file on the remote server, that more than just the changed bits are being copied over.  I say this because I use a VPN connection between the two and it tells me 2+ GB has passed through the connection and that's about the size of my .bkf file.  However, it does seem to happen a lot faster than simple XCopy so maybe I need to do some experimenting and see what's going on for sure, ie. whether I'm actually benefiting from rsync delta copies or if rsync is just inherently faster at copying than XCopy without delta being a factor.  Do you use rsync?  Do people always combine it with some other method (such as NTBackup) to make it effective for backup?

Thanks.
0
 
LVL 88

Expert Comment

by:rindi
ID: 17856093
I haven't used rsync yet, but it is on my list for testing when I can get the chance, as it is supposed to be perfect. I never use xcopy though, rather robocopy, which is a free commandline utility in the windows 2000 server resource kit tools. To get the system state you do need a dedicated software like ntbackup.
0
 
LVL 8

Assisted Solution

by:Disorganise
Disorganise earned 250 total points
ID: 17871509
I imagine the reason rsync appears faster is because it's been optimised for WAN communication, whereas Xcopy is very locally oriented.  ie, the number of confirmation and ordering packets created by xcopy would be significantly larger than the number generated by rsync I would imagine.  it's the same argument for why WAFS speeds up WAN file performance.

0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 18074600
Sorry about that, didn't close the question because I never got a good answer.  But I appreciate the responses and agree with the recommendations for a split. Thanks.
0

Featured Post

Give your grad a cloud of their own!

With up to 8TB of storage, give your favorite graduate their own personal cloud to centralize all their photos, videos and music in one safe place. They can save, sync and share all their stuff, and automatic photo backup helps free up space on their smartphone and tablet.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
VMWare 6.0 Trial not finding Logical drives on MSA70 via P800 28 101
DGTEC 10.1" Quad Core Android Tablet 18 95
File "Archiving" 5 58
File copy time stamp question 6 54
Lets start to have a small explanation what is VAAI(vStorage API for Array Integration ) and what are the benefits using it. VAAI is an API framework in VMware that enable some Storage tasks. It first presented in ESXi 4.1, but only after 5.x sup…
A Bare Metal Image backup allows for the restore of an entire system to a similar or dissimilar hardware. They are highly useful for migrations and disaster recovery. Bare Metal Image backups support Full and Incremental backups. Differential backup…
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…
This tutorial will show how to configure a single USB drive with a separate folder for each day of the week. This will allow each of the backups to be kept separate preventing the previous day’s backup from being overwritten. The USB drive must be s…

895 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now