Solved

RSync & NTBackup dilemma - compression

Posted on 2006-10-30
7
1,761 Views
Last Modified: 2013-12-01
The question: How can I make the most of rsync running under Windows and get it to store files in a compressed format?

The details:  I'm using rsync (DeltaCopy) and NTBackup to remotely backup one Windows 2003 server to another across broadband links. I'm looking for help from EE to try and determine the best method using the two apps or preferably just one.  Right now I'm doing nightly differential backups with NTBackup which includes the System State.  The resulting file is approx. 2GB and slowly growing.  It has about doubled in size in a month and I'm guessing it will continue to grow unless I do another full backup.  

I'm using NTBackup for the following reasons:
1 It gets the System State
2 It uses Shadow Copy to get all open files
3 It saves in one neat and easy file
4 COMPRESSION

Using rsync to get my .bkf file to the remote server sped things up significantly over XCopy.  It still says it's copying 2GB but I think it's telling me the entire size of the file it's copying, even if it only copies some of the bits.  It takes just under an hour to copy the 2GB over DSL/Cable lines.  

My biggest problem with using rsync alone is that I would miss out on the storage compression.  I know rsync compresses data to transfer over the wire but does it store the files on the remote end compressed?  I need to have it compressed on the remote server to save storage space.  I know about hardware compressing the drive, but isn't that additional compression beyond the software compression?  What I mean is, if I have a 2GB file and compress it to 1GB with NTBackup and then store it on a compressed drive, won't it still compress further?   I think with rsync, it may compress the 2GB file for transfer but I end up with a 2GB file on the other side (or ~1GB) on a compressed drive.  Is that correct?

Thanks in advance.
0
Comment
Question by:VortexAdmin
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
7 Comments
 
LVL 88

Accepted Solution

by:
rindi earned 250 total points
ID: 17848449
No, rsync just synchronizes the source with the destination. It won't compress the file on the destination. Anyway, isn't the bkf file you have created with ntbackup already compressed? A compressed file is usually not further compressable, or only to a very small extent which isn't worth the time. If the bkf file isn't compressed, then look at a commercial backup product for your server like veritas backupexec or arcserve which do compress the backup files.
0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 17851942
The .bkf is compressed, but I was trying to figure out a way not to have to use NTBackup at all. There would be some benefits to having the full copy on the remote server, instead of in a single .bkf file.  The reason I'm still using NTBackup though is because A) the files aren't compressed once they're on the remote server using just rsync and B) I need to be able to get the System State.

It looks to me like when a .bkf file is created on the source, and it includes the System State, then rsynced with the previous day's .bkf file on the remote server, that more than just the changed bits are being copied over.  I say this because I use a VPN connection between the two and it tells me 2+ GB has passed through the connection and that's about the size of my .bkf file.  However, it does seem to happen a lot faster than simple XCopy so maybe I need to do some experimenting and see what's going on for sure, ie. whether I'm actually benefiting from rsync delta copies or if rsync is just inherently faster at copying than XCopy without delta being a factor.  Do you use rsync?  Do people always combine it with some other method (such as NTBackup) to make it effective for backup?

Thanks.
0
 
LVL 88

Expert Comment

by:rindi
ID: 17856093
I haven't used rsync yet, but it is on my list for testing when I can get the chance, as it is supposed to be perfect. I never use xcopy though, rather robocopy, which is a free commandline utility in the windows 2000 server resource kit tools. To get the system state you do need a dedicated software like ntbackup.
0
 
LVL 8

Assisted Solution

by:Disorganise
Disorganise earned 250 total points
ID: 17871509
I imagine the reason rsync appears faster is because it's been optimised for WAN communication, whereas Xcopy is very locally oriented.  ie, the number of confirmation and ordering packets created by xcopy would be significantly larger than the number generated by rsync I would imagine.  it's the same argument for why WAFS speeds up WAN file performance.

0
 
LVL 5

Author Comment

by:VortexAdmin
ID: 18074600
Sorry about that, didn't close the question because I never got a good answer.  But I appreciate the responses and agree with the recommendations for a split. Thanks.
0

Featured Post

Simplifying Server Workload Migrations

This use case outlines the migration challenges that organizations face and how the Acronis AnyData Engine supports physical-to-physical (P2P), physical-to-virtual (P2V), virtual to physical (V2P), and cross-virtual (V2V) migration scenarios to address these challenges.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Google photos for iphone 6 66
EqualLogic PS3000 Switch Port Setup 1 46
MySQL 5.6.30 - daily outages 46 61
Hyper-v migrate to new storage 1 64
Workplace bullying has increased with the use of email and social media. Retain evidence of this with email archiving to protect your employees.
The question appears often enough, how do I transfer my data from my old server to the new server while preserving file shares, share permissions, and NTFS permisions.  Here are my tips for handling such a transfer.
This tutorial will walk an individual through the steps necessary to install and configure the Windows Server Backup Utility. Directly connect an external storage device such as a USB drive, or CD\DVD burner: If the device is a USB drive, ensure i…
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…

756 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question