Yargatech
asked on
Rsync Issues
Hey all,
I am having a bad Rsync issue and while I am pretty sure I know the answer (ie I'm screwed) I wanted to bounce if off the experts first.
I started using a D2D backup solution with a local readynas as the backup target.
Everything worked great for the first month and then everything has fallen apart. I thought the backup worked a bit differently and I think that assumption has caused my solution to be a failure.
The way the backup works is that each day it creates a new folder, well a number of folders, but only one is important due to it having the actual data files (the bulk of the data). So the files are structured as:
S000001
S000002
S000003
S000004
Etc.
Inside each one of these folders is a file containing the backed up data for the drive. Each filename is the same such as:
Drive01234.D2D
The problem is once you reach the maximum number of restore points.
What happens at this point is that the oldest full backup is merged into the incremental. So for the example above the .D2D file in S00001 is merged with the .D2D in S0002
Then this merged file is saved ins S00002 and S00001 is purged. This continueS each day.
The problem then is that Rsync sees the file in S00002 as the incremental + FUll, rather than Full + incremental.
So for example if my file in S00001 was 14GB, and the incremental in S00002 was 1GG, the merged file according to Rsync is 14GB of changed bits rather than the reverse.
The end result is that after 30 days of backups, Rsync is basically trying to sync a full backup every night, and bogs down.
I am assuming there is probably nothing I can do about this, but am hoping I am wrong.
I am having a bad Rsync issue and while I am pretty sure I know the answer (ie I'm screwed) I wanted to bounce if off the experts first.
I started using a D2D backup solution with a local readynas as the backup target.
Everything worked great for the first month and then everything has fallen apart. I thought the backup worked a bit differently and I think that assumption has caused my solution to be a failure.
The way the backup works is that each day it creates a new folder, well a number of folders, but only one is important due to it having the actual data files (the bulk of the data). So the files are structured as:
S000001
S000002
S000003
S000004
Etc.
Inside each one of these folders is a file containing the backed up data for the drive. Each filename is the same such as:
Drive01234.D2D
The problem is once you reach the maximum number of restore points.
What happens at this point is that the oldest full backup is merged into the incremental. So for the example above the .D2D file in S00001 is merged with the .D2D in S0002
Then this merged file is saved ins S00002 and S00001 is purged. This continueS each day.
The problem then is that Rsync sees the file in S00002 as the incremental + FUll, rather than Full + incremental.
So for example if my file in S00001 was 14GB, and the incremental in S00002 was 1GG, the merged file according to Rsync is 14GB of changed bits rather than the reverse.
The end result is that after 30 days of backups, Rsync is basically trying to sync a full backup every night, and bogs down.
I am assuming there is probably nothing I can do about this, but am hoping I am wrong.
ASKER
Keep in mind its not my backup. It is a software package I have been testing. So I don't have control over the way it works.
It does seem to keep the timestamp. But in my opinion it merges backwards in order to function properly with Rsync.
To kind of explain differently it merges as such
Before backup you would have
S0000001 folder
------Disk1234.D2D size 14GB
S0000002 folder
------Disk1234.D2D size 1GB
After the backup
S0000001 folder is deleted.
S0000002 folder
-----Disk1234.D2D size 14GB. This is the file merged with S0000001.
Yes the timestamp seems to be the same. But according to Rsync the file has increased 13GB. I don't see any way that rsync can know this is the same file moved from folder S0000001.
In my opinion when they merge the file it should always be merging into the S000001 folder so that rsync would see the file as having only had 1GB of changes.
I don't see anyway around this other than finding a different way to sync or a different backup program.
It does seem to keep the timestamp. But in my opinion it merges backwards in order to function properly with Rsync.
To kind of explain differently it merges as such
Before backup you would have
S0000001 folder
------Disk1234.D2D size 14GB
S0000002 folder
------Disk1234.D2D size 1GB
After the backup
S0000001 folder is deleted.
S0000002 folder
-----Disk1234.D2D size 14GB. This is the file merged with S0000001.
Yes the timestamp seems to be the same. But according to Rsync the file has increased 13GB. I don't see any way that rsync can know this is the same file moved from folder S0000001.
In my opinion when they merge the file it should always be merging into the S000001 folder so that rsync would see the file as having only had 1GB of changes.
I don't see anyway around this other than finding a different way to sync or a different backup program.
Can it rename back to S0000001, such as
$ mv S0000002 S0000001
after merge?
$ mv S0000002 S0000001
after merge?
ASKER
I don't see any easy way to rename without affecting the potential integrity of the backup program. Every day it increments again. So once it merges 1 with 2 it deletes 1. Then the next day it merges 2 with 3 and deletes 2. I would have to have some sort of script to rename the folders at both ends prior to the rsync backup, and then rename back when it is complete.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
When you merge, do you preserve the timestamp of files?
rsync checks the timestamp of files first.