We help IT Professionals succeed at work.

Check out our new AWS podcast with Certified Expert, Phil Phillips! Listen to "How to Execute a Seamless AWS Migration" on EE or on your favorite podcast platform. Listen Now


Rsync Issues

Yargatech asked
Medium Priority
Last Modified: 2012-05-11
Hey all,

I am having a bad Rsync issue and while I am pretty sure I know the answer (ie I'm screwed) I wanted to bounce if off the experts first.

I started using a D2D backup solution with a local readynas as the backup target.

Everything worked great for the first month and then everything has fallen apart.  I thought the backup worked a bit differently and I think that assumption has caused my solution to be a failure.

The way the backup works is that each day it creates a new folder, well a number of folders, but only one is important due to it having the actual data files (the bulk of the data).  So the files are structured as:



Inside each one of these folders is a file containing the backed up data for the drive.  Each filename is the same such as:


The problem is once you reach the maximum number of restore points.

What happens at this point is that the oldest full backup is merged into the incremental.  So for the example above the .D2D file in S00001 is merged with the .D2D in S0002

Then this merged file is saved ins S00002 and S00001 is purged.  This continueS each day.

The problem then is that Rsync sees the file in S00002 as the incremental + FUll, rather than Full + incremental.

So for example if my file in S00001 was 14GB, and the incremental in S00002 was 1GG, the merged file according to Rsync is 14GB of changed bits rather than the reverse.

The end result is that after 30 days of backups, Rsync is basically trying to sync a full backup every night, and bogs down.

I am assuming there is probably nothing I can do about this, but am hoping I am wrong.
Watch Question

Top Expert 2011

It sounds that the way you merge causing the problem.

When you merge, do you preserve the timestamp of files?
rsync checks the timestamp of files first.


Keep in mind its not my backup.  It is a software package I have been testing. So I don't have control over the way it works.

It does seem to keep the timestamp.  But in my opinion it merges backwards in order to function properly with Rsync.

To kind of explain differently it merges as such

Before backup you would have

S0000001 folder
------Disk1234.D2D size 14GB
S0000002 folder
------Disk1234.D2D size 1GB

After the backup

S0000001 folder is deleted.

S0000002 folder
-----Disk1234.D2D size 14GB.  This is the file merged with S0000001.

Yes the timestamp seems to be the same.  But according to Rsync the file has increased 13GB.  I don't see any way that rsync can know this is the same file moved from folder S0000001.

In my opinion when they merge the file it should always be merging into the S000001 folder so that rsync would see the file as having only had 1GB of changes.

I don't see anyway around this other than finding a different way to sync or a different backup program.
Top Expert 2011

Can it rename back to S0000001, such as
$ mv S0000002 S0000001
after merge?


I don't see any easy way to rename without affecting the potential integrity of the backup program.  Every day it increments again.  So once it merges 1 with 2 it deletes 1.  Then the next day it merges 2 with 3 and deletes 2.  I would have to have some sort of script to rename the folders at both ends prior to the rsync backup, and then rename back when it is complete.
Top Expert 2011
Unlock this solution and get a sample of our free trial.
(No credit card required)
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.