looking for a bit or block level replication or backup software

I know Rsync does bit level syncs of data and can be automated via scheduled tasks, is free and easy to use.

have a read at: http://samba.anu.edu.au/rsync/

You could try SyncOnDemand. It will do real time data transfers. http://www.caddais.com/Products.html

Also for making images of windows systems I use Drivesnapshot http://drivesnapshot.de/en/index.htm

It will do differential backups.

Hope this helps.

Mr-Madcowz,

I had a look at a windows gui version of rsync called deltacopy before my question and i didn't get it to work straight away but I'l look into rsync again and a bit deeper if needed

TCB1,

I had a look at SyncOnDemand but couldn't find any reference of delta or partial or bit level or block level copying/syncing so it seems your solution does not meet the requirements (unless i missed something)

You are correct, SyncOnDemand just copies the files themselves. However, I was thinking you could make an image of your system (at how often you need) with DriveSnapShot which does do differential image backups. So you would only be copying the changes over your WAN link. Only the initial image file of the system would take a while to transfer. But after that, the changes would transfer quickly. You can set the filesize you want to split the image files into. (ie. transfer 10-50MB files instead of one 500MB file)

Then once you make the image files, use SyncOnDemand to transfer the image files to the target via the WAN link. You could setup SyncOnDemand to transfer in real time or by schedule.

The end result would be a copy of your images directory on the target, only copying the changes.
I've never tried this on such a large scale, so I would do lots of testing before I decided to commit. In theory it should work.

TCB1,

I see your larger plan and I did interpret it that way but that's not my goal. The reason is that I've tried this method long ago in the past and it gave me 2 problems:

1. Restore took too much time since it had to happen over a wan link and full images need to be taken into account as well including incrementals to restore

2. I had no full redundancy. I had the incrementals either on  site or over the WAN link (offsite) but if something happened to that I had nothing left.

Therefore my big plan is to dump images locally which allows for blazing fast recovery at any time and then either backup those images to tape and take some offsite or have them all time synced to an offsite location which would be ideal. And that is where this question comes in.

ps: Acronis does incrementals as well.

ps: one could argue to simply schedule multiple jobs, one to local nas and one to offsite but again that's not my plan in this question

Keep it coming guys. In the meanwhile I'll give rscync/deltacopy a try

Do persevere with Rsync.

We use it here in many ways.

Every day I snapshot my data attached to the Primary Server on the San and then use Rsync to mirror it over to the LUN attached to the secondary server.

We also use it to copy data over the VPN to our other site and also to keep system files synched on all our 3 application servers.

Once set up it just runs and also produces nice logs which I always like.

im not sure if it can do it, but take a look at symantec exec

Mr-Madcowz,

have you got any links on handy scripts or examples in using it ? I'll be trying it out tomorrow

There are several different version. A lot of people swear by Deltacopy but I didn't have any luck with that. I just used rsync and the two cygwin dlls that you drop into %systemroot%/system32

If you are wanting to provide secure sync you should also plumb in the ssh side of things as well.

I am trying to find the definitive documents that I used to set up my systems but I can't find them. I will continue to hunt around. Meanwhile, here are some links that I have bookmarked. Whether they were part of my install I can't remember, sorry.

http://brentnorris.net/rsyncntdoc.html
http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html
http://www.itefix.no/i2/node/10650
http://www.exavault.com/support-rsync-setup-windows.php

I looked at Deltacopy again but found that it requires a servercomponent to be installed on 1 side and a client executed on the other side. However, as mentioned in the original question I cannot execute anything on the other side, I can only access it as \\nas01\sharename and it doesn't even run windows. so i guess Deltacopy is not a solution. Right now I downloaded cwRsync (the windows version) and I'm testing as we speak. i'm able to copy a large pst file from c:\test1 to c:\test2 but after modifying the pst file in c:\test1 (adding a 3mb email in inbox) and then rsyncing it seems to copy the entire file again and not just the changes. I suppose I must use some specific switches wich I havent' done yet so far. My current command was:

rsync -r /cygdrive/c/test1/ /cygdrive/c/test2/

I have yet to get to work:

1. only delta changes being copied
2. do the copy not to a local second dir but a shared dir on the nas

Ok,

I have just installed this on a client desktop and am now syncing his files to a mapped drive on our server. Ok this isn't using ssh or running it as a daemon but it does show how quickly a sync can be setup. Total setup time less than 5 minutes.

I installed:

cygpopt-0.dll
cygwin1.dll

into %systemroot%\system32

and then placed rysnc.exe into c:\program files\rsync

(all these files are available in a zip at: http://www.brentnorris.net/rsyncntdoc.html

You will need to add a path to this dir if you want to be able to run rsync from anywhere on the system, but since I always run it from a batch script via a scheduled task, I just call it from it's root.

I then create a batch script. For example:

rsync -avp --delete /cygdrive/G/SAN_ARRAY/XPP_Data/alljobz/  "/cygdrive/F/SAN_ARRAY/XPP_Data/alljobz/" > rsync_log_alljobz.txt

This syncs from G:\SAN_ARRAY\XPP_Data\alljobz to F:\SAN_ARRAY\XPP_Data\alljobz and writes to the log file called rsync_log_alljobz.txt

The cygdrive is used because it doesn't understand the windows C:\ way of pathing and the switches are

-a = archive mode
-v = verbose
-p = preserve permissions
--delete = delete files on target that are not on source (i.e. mirror it)

You can get the full man page from here:

http://www.samba.org/ftp/rsync/rsync.html

As I said, this is a quick and dirty non secure way of synching between two mapped drives. It is a little more involved if you want to set up an rsync server and ssh, but still pretty simple.

Using your line I sucessfully synced my c:\test1 with a shared O:\ but the amount of time required seemed to be as long as if the entire copy had to happen on not just the changed part (3MB) of the .pst file. Therefore it looks like the entire file is simply copied over. Not just the 3MB. Is there any way I can verify or prove that only the delta of the file is being copied over ?

AHA! It seems the key to this problem is a bit hidden between the lines in the MAN:

one line reads:

"If any of the files already exist on the remote system then the rsync remote-update protocol is used to update the file by sending only the differences. See the tech report for details. "

while another reads:

"You can also use rsync in local-only mode, where both the source and destination don't have a ':' in the name. In this case it behaves like an improved copy command"

apparently these 2 lines hit that delta copy only works when you specifically use a remote while copying to a local driveletter (which is in fact a mapping to a shared dir on another device) does not work with delta.

If that is the requirement another line reads:

"You can also specify any remote shell you like, either by using the -e command line option, or by setting the RSYNC_RSH environment variable. "

"Note that rsync must be installed on both the source and destination machines"

If I read this correctly, it would mean that delta copying does not provide the solution in this case since delta copying only works to remote systems where you can actually install and run the remote rsync. In this case is it not possible to install software on the nas. That also implies that delta copy will not work and rsync works in this case no better than a normal copy command. It will still try to completely copy 50GB files entirely over the wan link.

Upon digging deeper on this issue, a question from the past seems to confirm these findings:

https://www.experts-exchange.com/questions/22976845/delta-copy-for-rsync.html

wow,

I guess there really isn't a working piece of software on this planet that can provide the result (calculating all restriction in)

Let me test Rsync again today on two brand new untouched (Rsync wise) machines as I am sure that it only needs to be installed at one end (the source).

Rsync definately does only copy the changes as I sync 240Gb of data in less than 4 hours every day and there is no way it can just be copying that all over and the log reports also state this witht he total byte size being less than the actual amount on disk.

thx Madcowz,

I'll be eagerly awaiting yoru response. I did the test with installlation on only one end and using a shared/mapped drive on the remote nas but I always get full copies according to the logs and time needed for copying. It really seems one needs to be able to install and rsync as well on the remote side to get it running. That is not possible with my current nas. I would have to buy one that has native support for it such as the Netgear readynas but that's out of the scope of this question.

I hope you find it to be supported

I have just tested rsync with a 571Mb zip file containing our Backup Exec install.

First I copied it into both source and target and ran an rsysnc.

building file list ...
 0 files...
2 files to consider
./

Number of files: 2
Number of files transferred: 0
Total file size: 584742597 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 118
File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 146 Total bytes received: 32

sent 146 bytes  received 32 bytes  356.00 bytes/sec total size is 584742597  speedup is 3285070.77


Fair enough I thought.

Then I opened up the zip and added the word FART to a css file. (Yea, I know childish but I find that it is a good word to use as it won't be anywhere else in the file) and then ran another rsync.

building file list ...
 0 files...
2 files to consider
./
BackUpExec-11D.7170_32BIT_VERSION.zip
       32768   0%    0.00kB/s    0:00:00
    16482304   2%   15.67MB/s    0:00:35
    32735232   5%   15.59MB/s    0:00:34
    50429952   8%   16.02MB/s    0:00:32
    66846720  11%   15.93MB/s    0:00:31
    81887232  14%   15.59MB/s    0:00:31
    98074624  16%   15.57MB/s    0:00:30
   110133248  18%   14.14MB/s    0:00:32
   118128640  20%   12.14MB/s    0:00:37
   135299072  23%   12.64MB/s    0:00:34
   149061632  25%   12.07MB/s    0:00:35
   164102144  28%   12.85MB/s    0:00:31
   181764096  31%   15.16MB/s    0:00:25
   198770688  33%   15.12MB/s    0:00:24
   216334336  36%   16.02MB/s    0:00:22
   235372544  40%   16.98MB/s    0:00:20
   244023296  41%   14.83MB/s    0:00:22
   263716864  45%   15.48MB/s    0:00:20
   280264704  47%   15.24MB/s    0:00:19
   297828352  50%   14.79MB/s    0:00:18
   316997632  54%   17.28MB/s    0:00:15
   333611008  57%   16.53MB/s    0:00:14
   349667328  59%   16.42MB/s    0:00:13
   364249088  62%   15.82MB/s    0:00:13
   381976576  65%   15.47MB/s    0:00:12
   394887168  67%   14.59MB/s    0:00:12
   413597696  70%   15.22MB/s    0:00:10
   429490176  73%   15.54MB/s    0:00:09
   446267392  76%   15.31MB/s    0:00:08
   464551936  79%   16.59MB/s    0:00:07
   476577792  81%   15.00MB/s    0:00:07
   495681536  84%   15.77MB/s    0:00:05
   511803392  87%   15.62MB/s    0:00:04
   529793024  90%   15.55MB/s    0:00:03
   546209792  93%   16.60MB/s    0:00:02
   563937280  96%   16.27MB/s    0:00:01
   581697536  99%   16.66MB/s    0:00:00
   584742598 100%   15.39MB/s    0:00:36  (1, 100.0% of 2)


   
Number of files: 2
Number of files transferred: 1
Total file size: 584742598 bytes
Total transferred file size: 584742598 bytes
Literal data: 584742598 bytes
Matched data: 0 bytes
File list size: 118
File list generation time: 0.002 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 584814160
Total bytes received: 48
sent 584814160 bytes  
received 48 bytes  
15595045.55 bytes/sec
total size is 584742598  



So, this looks to me as though it is transferring the entire file and I am now confused.

Everything I read says that Rsync copies at byte level...

Puzzled.

I have just posted my log output to the rsync mailing list and will advise you of what comes back.

I think it's litteraly performing the way the man states: if you use a remote (installed and running rsync on the other side) it can use "update mode" which  means byte level copy. But if you use it "locally" because e.g you can't install software on the remote side, it becomes a normal advanced copy program like robocopy without delta. In short: every line that contains rsync withouta the character ":" in it will not do delta copies, only normals. I hope it's not that way though

Right then.

I got a response from the mailing list and it states:

"Locally used rsync will default to -W which causes whole files to be transferred if they differ..."

-W is the same as --whole-file and just below that is the --no-whole-file option.

So I tested that and got these results.


Number of files: 2
Number of files transferred: 1
Total file size: 584742599 bytes
Total transferred file size: 584742599 bytes
Literal data: 46039 bytes
Matched data: 584696560 bytes
File list size: 118
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 142969
Total bytes received: 169357


   sent 142969 bytes  
   received 169357 bytes  
   
   6064.58 bytes/sec

   total size is 584742599  speedup is 1872.22


So, that is a big difference in the total bytes sent.  From 584814160 with the --whole-file option to 142969 with the --no-whole-file option.

My only concern now is that I was editing a .css file within a zip and was it sending the entire .css file or just the changes? The true test would be to have a large single file, say a .exe and edit a few bytes in that and then test.

But at least in this case we have a local rsync going from transferring 557.7222Mb down to only 0.1363Mb with a simple switch.

this is very interesting: a --no-whole-file option, especially since that option doesn't appear anywhere in my version.
So I'm doublecheck wether this is working for me:

I mapped driveletter O: to a shared volume containing 1 big .pst of around 430MB and then sync it to c:\test1. First initial, then again after adding a 3MB email to the PST file, then again after nothing and again after adding another 3MB email. Here are the results:

sending incremental file list
./
archive1.pst
~archive3.pst.tmp

sent 442505158 bytes  received 53 bytes  8119361.67 bytes/sec
total size is 445936640  speedup is 1.01
 (this was the initial copy)


sending incremental file list
./
archive1.pst

sent 9579379 bytes  received 146712 bytes  150792.11 bytes/sec
total size is 452031488  speedup is 46.48
 (after adding 3MB email)


sending incremental file list

sent 121 bytes  received 12 bytes  266.00 bytes/sec
total size is 452031488  speedup is 3398732.99
 (no update)


sending incremental file list
./
archive1.pst

sent 6676923 bytes  received 147734 bytes  95449.75 bytes/sec
total size is 454063104  speedup is 66.53
 (+3MB again)

according to this log file, that does indeed do the trick! You deserve the points definately and they will feel 3 times as good as many other points you got :)

However and totally optional to this question, I'm puzzled by 2 things:

1. the --no-whole-file option is not in my list (I don't care, as long as it does work)
2. The total time needed for each operation in my log file is the exact same 50 seconds, why ??!! I was expecting that delta copy would be faster sinc it only has to transfer 10MB instead of a 410MB. Yet the total time needed (timed with chronometer) is the exact same amount

If I time the seconds needed (about 50) each operation requires the same amount of time. I expected it to go faster since it's not supposed to transfer the whole file again.

here's what I get when I type rsync:

rsync  version 3.0.4  protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace,
    append, ACLs, no xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see manpage caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        make backups into hierarchy based in DIR
     --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
     --inplace               update destination files in-place (SEE MAN PAGE)
     --append                append data onto shorter files
     --append-verify         like --append, but with old data in file checksum
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
     --safe-links            ignore symlinks that point outside the source tree
 -k, --copy-dirlinks         transform symlink to a dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 -E, --executability         preserve the file's executability
     --chmod=CHMOD           affect file and/or directory permissions
 -A, --acls                  preserve ACLs (implies --perms)
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve modification times
 -O, --omit-dir-times        omit directories from --times
     --super                 receiver attempts super-user activities
 -S, --sparse                handle sparse files efficiently
 -n, --dry-run               perform a trial run with no changes made
 -W, --whole-file            copy files whole (without delta-xfer algorithm)
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
     --rsync-path=PROGRAM    specify the rsync to run on the remote machine
     --existing              skip creating new files on receiver
     --ignore-existing       skip updating files that already exist on receiver
     --remove-source-files   sender removes synchronized files (non-dirs)
     --del                   an alias for --delete-during
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer, not during
     --delete-during         receiver deletes during transfer (default)
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not during
     --delete-excluded       also delete excluded files from destination dirs
     --ignore-errors         delete even if there are I/O errors
     --force                 force deletion of directories even if not empty
     --max-delete=NUM        don't delete more than NUM files
     --max-size=SIZE         don't transfer any file larger than SIZE
     --min-size=SIZE         don't transfer any file smaller than SIZE
     --partial               keep partially transferred files
     --partial-dir=DIR       put a partially transferred file into DIR
     --delay-updates         put all updated files into place at transfer's end
 -m, --prune-empty-dirs      prune empty directory chains from the file-list
     --numeric-ids           don't map uid/gid values by user/group name
     --timeout=SECONDS       set I/O timeout in seconds
     --contimeout=SECONDS    set daemon connection timeout in seconds
 -I, --ignore-times          don't skip files that match in size and mod-time
     --size-only             skip files that match in size
     --modify-window=NUM     compare mod-times with reduced accuracy
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
     --compare-dest=DIR      also compare destination files relative to DIR
     --copy-dest=DIR         ... and include copies of unchanged files
     --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with a suffix in LIST
 -C, --cvs-exclude           auto-ignore files the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN       don't exclude files matching PATTERN
     --include-from=FILE     read include patterns from FILE
     --files-from=FILE       read list of source-file names from FILE
 -0, --from0                 all *-from/filter files are delimited by 0s
 -s, --protect-args          no space-splitting; only wildcard special-chars
     --address=ADDRESS       bind address for outgoing socket to daemon
     --port=PORT             specify double-colon alternate port number
     --sockopts=OPTIONS      specify custom TCP options
     --blocking-io           use blocking I/O for the remote shell
     --stats                 give some file-transfer stats
 -8, --8-bit-output          leave high-bit chars unescaped in output
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer
 -P                          same as --partial --progress
 -i, --itemize-changes       output a change-summary for all updates
     --out-format=FORMAT     output updates using the specified FORMAT
     --log-file=FILE         log what we're doing to the specified FILE
     --log-file-format=FMT   log updates using the specified FMT
     --password-file=FILE    read daemon-access password from FILE
     --list-only             list the files instead of copying them
     --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
     --write-batch=FILE      write a batched update to FILE
     --only-write-batch=FILE like --write-batch but w/o updating destination
     --read-batch=FILE       read a batched update from FILE
     --protocol=NUM          force an older protocol version to be used
     --iconv=CONVERT_SPEC    request charset conversion of filenames
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h works with no other options)

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at main.c(1408) [client=3.0.4]

I'm using rsync  version 2.6.6  protocol version 29 so we have different versions and it seems that yours (the newer version) doesn't mention that switch.

Regarding the time taken, I can only assume that the time taken is for the app to compare the two files rather than transfer data. Beyond that I don't know. I may well post both these points to the mailing list.

Thanks for those points, they do indeed feel good.

You rock! thanks for such persistence to get to the bottom of things and dig up the dirt. That switch does the trick indeed. I for one would very much appreciate if you get get an answer to that last question about time taken and post it here. I did think about the time needed for compare as well to be honest and considering the fact that I do my test to a shared volume on a local NAS and not on a NAS behind a wan link could make for a less obvious difference in time than the same operation over a WAN link. However, if comparing of the 2 files is done in-place over the WAN link, will it not require roughly the same amount of traffic over the WAN link (to request each bit for inspection) as if one had just copied over the entire file again instead of only the delta ? That would make it still truly only sending Delta but not necesseraly give a decrease in time or traffic needed to copy files over a wan link...right ?