rsync: Daily, Weekly and Monthly backups

I want to use rsync to store daily, weekly and monthly backups of my /home/ directory on my production server to a backup server.

I want seven daily backups to be kept.  So every day, the oldest daily backup should be deleted and a new backup added.

I want five weekly backups to be kept.  So every week, the oldest weekly backup should be deleted and the newest daily backup should be kept as a weekly backup.

I want twelve monthly backups to be kept.  So every month, the oldest monthly backup should be deleted and the newest daily backup should be kept as a weekly monthly backup.

LVL 16
hankknightAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

IanThCommented:
there are a number of linux backup tools available are you sure you want to use rsync
0
hankknightAuthor Commented:
I am using an online service that only supports rsync.
0
IanThCommented:
the usual way is to use cron for scheduling
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

robthewolfCommented:
you put a line in cron for each different backup that you want.
one for each day of the week, one for the week and one for the month.
each line in the cron will write to the specific backup that you want.
For example you will have a line that backs up on a monday.  It will always back up to the same place, which will be designated at the monday backup.  This will happen once a week and the following week will update the previous monday backup
0
arober11Commented:
Following on from _robthewolf_ suggestion:

First you need to generate a pass phrase less a [DR]SA key pair, to allow password less ssh / rsync access to the remote host.  e.g. type:

ssh-keygen -t rsa

and just keep hitting <enter>.

THen on the remote server create the following directory structure somewhere:
/backup/Mon/
/backup/Tue/
/backup/Wed/
...
/backup/Sun/

/backup/Week1/
/backup/Week2/
/backup/Week3/
/backup/Week4/
/backup/Week5/

/backup/Jan/
/backup/Feb/
/backup/Mar/
...
/backup/Dec/

Then on the local server add something like the following to your users crontab e.g.  

# BAckup $HOME to a remote server
# Daily backup
05 02 * * * /xxx/xxx/bin/rsync -optlgvr --delete-after --rsh=/usr/bin/ssh --rsync-path=/REMOTE_RSYNC_BIN_DIR/rsync --force "$HOME/"  "user@remote.host.com:/xxx/backup/`date '+\%b'`" > /tmp/backup_daily 2>&1
#
# Weekly backup
05 03 * * 1  /xxx/xxx/bin/rsync -optlgvr --delete-after --rsh=/usr/bin/ssh --rsync-path=/REMOTE_RSYNC_BIN_DIR/rsync --force "$HOME/"  "user@remote.host.com:/xxx/backup/Week$((`date '+\%V'` % 5))" > /tmp/backup_weekly 2>&1
#
# An alternative approach to do the same weekl backup:
#05 03 * * 1  dailyBackup="/xxx/backup/`date '+\%b'`";newBackup="/xxx/backup/Week$((`date '+\%V'` \% 5))"; ssh user@remote.host.com "rm -Rf $newBackup && cp -Rp $dailyBackup $newBackup"  > /tmp/backup_weekly 2>&1
#
# Monthly backup
05 04 1 * * /xxx/xxx/bin/rsync -optlgvr --delete-after --rsh=/usr/bin/ssh --rsync-path=/REMOTE_RSYNC_BIN_DIR/rsync --force "$HOME/"  "user@remote.host.com:/xxx/backup/`date '+\%a'`" > /tmp/backup_monthly 2>&1
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
bman21Commented:
just to expand on robthewolf and IanTh's suggestion, the cron daemon would have to look something like this:
####################################################
# Daily backups: This will run rsync every night at                      #
#  1:00 a.m. (you can change the time to whatever  you want)  #
####################################################
#Sunday
0 1 * * 0 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/SUNDAY/

#Monday
0 1 * * 1 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/MONDAY/

#Tuesday
0 1 * * 2 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/TUESDAY/

#Wednesday
0 1 * * 3 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/WEDNESDAY/

#Thursday
0 1 * * 4 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/THURSDAY/

#Friday
0 1 * * 5 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/FRIDAY/

#Saturday
0 1 * * 6 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/SATURDAY/

##############################
# Keep 4 WEEKLY "snapshots"        #
# runs at 2 am Saturdays if the day #
# of the month falls on the given       #
# day range                                       #
##############################

#Week 1
0 2 1-7 * 6 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/WEEK_1/

#Week 2
0 2 8-14 * 6 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/WEEK_2/

#Week 3
0 2 15-21 * 6 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/WEEK_3/

#Week 4
0 2 22-31 * 6 rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/WEEK_4/

#################
# Monthly backups  #
#################

#January
0 3 * 1 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/January/

#February
0 3 * 2 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/February/

#March
0 3 * 3 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/March/

#April
0 3 * 4 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/April/

#May
0 3 * 5 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/May/

#June
0 3 * 6 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/June/

#July
0 3 * 7 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/July/

#August
0 3 * 8 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/August/

#September
0 3 * 9 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/September/

#October
0 3 * 10 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/October/

#November
0 3 * 11 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/November/

#December
0 3 * 12 * rsync -avu --delete /LOCAL/DIRECTORY/ -e ssh remote_server:/REMOTE/DIRECTORY/December/

##################
# End of Crontab file #
##################

**NOTE** you can use the --dry-run option to ensure it is doing what you want to do without actually copying anything over.  

**NOTE** just for clarity, the * * * * * in front of the rsync command tells cron when to run rsync.  Each * stands for MINUTES, HOUR, DAY_OF_MONTH, MONTH, DAY_OF_WEEK.  

**NOTE**  I am not to sure if the weekly scripts will work.  Technicaly, the Daily scropts we created is the way to do weekly cron jobs, but since you are wanting to keep a weekly backup, that is the only way that i know how to do it short of actually creating a shell script.

The Daily and Monthly scripts will work though.  Notice that I added the "-u" and "--delete" commands.  The -u, or update, switch saves time on your backups after the first one has been created.  The --delete will delete any files off the remote directory that are no longer existent on the local filesystem.

I know that was a lot to dump on you.  If you have any questions please feel free to ask.  The above written code should be a "plug and play" minus a few alterations to the actually directory and the ssh connection to your backup site.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.