Solved

RSync & NTBackup dilemma - compression

Posted on 2006-10-30
7
1,760 Views
Last Modified: 2013-12-01
The question: How can I make the most of rsync running under Windows and get it to store files in a compressed format?

The details:  I'm using rsync (DeltaCopy) and NTBackup to remotely backup one Windows 2003 server to another across broadband links. I'm looking for help from EE to try and determine the best method using the two apps or preferably just one.  Right now I'm doing nightly differential backups with NTBackup which includes the System State.  The resulting file is approx. 2GB and slowly growing.  It has about doubled in size in a month and I'm guessing it will continue to grow unless I do another full backup.  

I'm using NTBackup for the following reasons:
1 It gets the System State
2 It uses Shadow Copy to get all open files
3 It saves in one neat and easy file
4 COMPRESSION

Using rsync to get my .bkf file to the remote server sped things up significantly over XCopy.  It still says it's copying 2GB but I think it's telling me the entire size of the file it's copying, even if it only copies some of the bits.  It takes just under an hour to copy the 2GB over DSL/Cable lines.  

My biggest problem with using rsync alone is that I would miss out on the storage compression.  I know rsync compresses data to transfer over the wire but does it store the files on the remote end compressed?  I need to have it compressed on the remote server to save storage space.  I know about hardware compressing the drive, but isn't that additional compression beyond the software compression?  What I mean is, if I have a 2GB file and compress it to 1GB with NTBackup and then store it on a compressed drive, won't it still compress further?   I think with rsync, it may compress the 2GB file for transfer but I end up with a 2GB file on the other side (or ~1GB) on a compressed drive.  Is that correct?

Thanks in advance.
0
Comment
Question by:VortexAdmin
  • 2
  • 2
7 Comments
 
LVL 88

Accepted Solution

by:
rindi earned 250 total points
ID: 17848449
No, rsync just synchronizes the source with the destination. It won't compress the file on the destination. Anyway, isn't the bkf file you have created with ntbackup already compressed? A compressed file is usually not further compressable, or only to a very small extent which isn't worth the time. If the bkf file isn't compressed, then look at a commercial backup product for your server like veritas backupexec or arcserve which do compress the backup files.
0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 17851942
The .bkf is compressed, but I was trying to figure out a way not to have to use NTBackup at all. There would be some benefits to having the full copy on the remote server, instead of in a single .bkf file.  The reason I'm still using NTBackup though is because A) the files aren't compressed once they're on the remote server using just rsync and B) I need to be able to get the System State.

It looks to me like when a .bkf file is created on the source, and it includes the System State, then rsynced with the previous day's .bkf file on the remote server, that more than just the changed bits are being copied over.  I say this because I use a VPN connection between the two and it tells me 2+ GB has passed through the connection and that's about the size of my .bkf file.  However, it does seem to happen a lot faster than simple XCopy so maybe I need to do some experimenting and see what's going on for sure, ie. whether I'm actually benefiting from rsync delta copies or if rsync is just inherently faster at copying than XCopy without delta being a factor.  Do you use rsync?  Do people always combine it with some other method (such as NTBackup) to make it effective for backup?

Thanks.
0
 
LVL 88

Expert Comment

by:rindi
ID: 17856093
I haven't used rsync yet, but it is on my list for testing when I can get the chance, as it is supposed to be perfect. I never use xcopy though, rather robocopy, which is a free commandline utility in the windows 2000 server resource kit tools. To get the system state you do need a dedicated software like ntbackup.
0
 
LVL 8

Assisted Solution

by:Disorganise
Disorganise earned 250 total points
ID: 17871509
I imagine the reason rsync appears faster is because it's been optimised for WAN communication, whereas Xcopy is very locally oriented.  ie, the number of confirmation and ordering packets created by xcopy would be significantly larger than the number generated by rsync I would imagine.  it's the same argument for why WAFS speeds up WAN file performance.

0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 18074600
Sorry about that, didn't close the question because I never got a good answer.  But I appreciate the responses and agree with the recommendations for a split. Thanks.
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

VM backup deduplication is a method of reducing the amount of storage space needed to save VM backups. In most organizations, VMs contain many duplicate copies of data, such as VMs deployed from the same template, VMs with the same OS, or VMs that h…
Are you looking to recover an email message or a contact you just deleted mistakenly? Or you are searching for a contact that you erased from your MS Outlook ‘Contacts’ folder and now realized that it was important.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question