Smart Sync or Binary Diff Tool

Posted on 2010-09-01
Medium Priority
Last Modified: 2012-05-10
We have the need to transfer copies of our builds to India for testing. These builds are about 4G in total but have huge amounts which do not change between files.
So the build contains a combination of many files and big CABs (or equivalents) . Some of the binaries might only change by a few bytes for the version stamping.

=> Is there a copy or sync tool out there which is really smart and can build remotely (either directly or by a copy and process) a copy of a file area based on a "seed" area. This seed area on the remote site would simply be a copy of the previous build ? It would need to recognise the common parts in the binary files.
Context : Windows Server 2003 / 2008

Is this essentially Microsofts DFS Replication ?

Now a normal sync tool would be file for the stand alone files that don;t change but hopeless for a very large binary which only has say 2% of the file changed. It would have no choice but to copy the large binary file.

Question by:greg_roberts
  • 3
  • 2

Accepted Solution

AndyEastham earned 375 total points
ID: 33596434

Have you looked a rsync?

Or for windows:

LVL 18

Expert Comment

ID: 33596493
Most international development team use some Repository tool ( like "SubVersioN" ) that allows :
-any "client" to replicate locally all (or a branch) of it (using CheckIn/CheckOut methods to avoid data write conflicts)
-versioning all files in the Repository
-a global Repository backup

You may although use a replication system like DFS to synchronize a directory in 2(+) locations but it requires sharing some Domain info

If your build does change all files at every new build, both Win2008 R2 DFS and SVN should allow for a differential/incremental evaluation...that is to be tested anyway
LVL 18

Expert Comment

ID: 33596521
SVN (SubVersioN)
On Windows, I have been using http://www.visualsvn.com/server (and its VisualSVN client) with success
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.


Author Comment

ID: 33601586
Thanks. i looked at rsync but could not run. No csyxxx.dll (some library dll to convert between windows to linux)

We use TFS 2080 for our source control and subversion is more of a source control tool than a replication tool.

Normal "sync" tools result in the 4G build needing 2G of b/w to copy. I am hoping that a block level trial will reduce this dramatically. Was trialing R1Soft but can't get the s/w to work. This has block level copy feature.
LVL 18

Assisted Solution

BigSchmuh earned 375 total points
ID: 33602245
My suggestion is to try using SVN (a source control tool) to replicate a directory locally:
-should allow for some incremental updates (Unsure with your binaries)

Author Closing Comment

ID: 33602265
Got rsync working using the url within the url for cwRsync but haven't got all the options right as i end up with the same sync transfer count as other tools (even using --inplace). Suspect issue with file times being used insead of CRC of file content.

As builds have complete binaries and files, exe, pdb, dlls , cab files etc i can't see how the SVN solution will help but thats for contrinuting.

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
Disk errors can be the source of sundry problems for the Exchange server, the most common one being that the database fails to mount.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question