Rsync Issues

Posted on 2011-04-21
Last Modified: 2012-05-11
Hey all,

I am having a bad Rsync issue and while I am pretty sure I know the answer (ie I'm screwed) I wanted to bounce if off the experts first.

I started using a D2D backup solution with a local readynas as the backup target.

Everything worked great for the first month and then everything has fallen apart.  I thought the backup worked a bit differently and I think that assumption has caused my solution to be a failure.

The way the backup works is that each day it creates a new folder, well a number of folders, but only one is important due to it having the actual data files (the bulk of the data).  So the files are structured as:



Inside each one of these folders is a file containing the backed up data for the drive.  Each filename is the same such as:


The problem is once you reach the maximum number of restore points.

What happens at this point is that the oldest full backup is merged into the incremental.  So for the example above the .D2D file in S00001 is merged with the .D2D in S0002

Then this merged file is saved ins S00002 and S00001 is purged.  This continueS each day.

The problem then is that Rsync sees the file in S00002 as the incremental + FUll, rather than Full + incremental.

So for example if my file in S00001 was 14GB, and the incremental in S00002 was 1GG, the merged file according to Rsync is 14GB of changed bits rather than the reverse.

The end result is that after 30 days of backups, Rsync is basically trying to sync a full backup every night, and bogs down.

I am assuming there is probably nothing I can do about this, but am hoping I am wrong.
Question by:Yargatech
    LVL 38

    Expert Comment

    It sounds that the way you merge causing the problem.

    When you merge, do you preserve the timestamp of files?
    rsync checks the timestamp of files first.

    Author Comment

    Keep in mind its not my backup.  It is a software package I have been testing. So I don't have control over the way it works.

    It does seem to keep the timestamp.  But in my opinion it merges backwards in order to function properly with Rsync.

    To kind of explain differently it merges as such

    Before backup you would have

    S0000001 folder
    ------Disk1234.D2D size 14GB
    S0000002 folder
    ------Disk1234.D2D size 1GB

    After the backup

    S0000001 folder is deleted.

    S0000002 folder
    -----Disk1234.D2D size 14GB.  This is the file merged with S0000001.

    Yes the timestamp seems to be the same.  But according to Rsync the file has increased 13GB.  I don't see any way that rsync can know this is the same file moved from folder S0000001.

    In my opinion when they merge the file it should always be merging into the S000001 folder so that rsync would see the file as having only had 1GB of changes.

    I don't see anyway around this other than finding a different way to sync or a different backup program.
    LVL 38

    Expert Comment

    Can it rename back to S0000001, such as
    $ mv S0000002 S0000001
    after merge?

    Author Comment

    I don't see any easy way to rename without affecting the potential integrity of the backup program.  Every day it increments again.  So once it merges 1 with 2 it deletes 1.  Then the next day it merges 2 with 3 and deletes 2.  I would have to have some sort of script to rename the folders at both ends prior to the rsync backup, and then rename back when it is complete.
    LVL 38

    Accepted Solution

    Well, in this case, rsync will not do any good to save time.
    Use gzip to compress and backup the gz file to save time and space.

    Featured Post

    Highfive + Dolby Voice = No More Audio Complaints!

    Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

    Join & Write a Comment

    This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
    The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
    Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
    This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now