Solved

Linux Script - Synchronize and keep incremental backup of directories/files

Posted on 2014-02-26
2
891 Views
Last Modified: 2014-03-13
Hi,
I have a site with very active development carried on day-to-day basis. I need to see what files my developer have worked on and need back them up for recovery.
The build of system runs in GB so I don't want to backup all files. Storage is seriously expensive.

I am looking for a linux script to
1. First backup changed files every hour and
2. Then all rsync files from master directory to secondary folder on same or remote server every hour.

Many Thanks
0
Comment
Question by:crazywolf2010
2 Comments
 
LVL 34

Accepted Solution

by:
Seth Simmons earned 250 total points
ID: 39890570
find . -mmin -60 | xargs -I% rsync -tgcop % /bar/%

Open in new window


generally speaking, you would use something like this
you find the files (replace . if you have a specific path and not using current folder)
the mmin -60 is the time specified; searching for anything modified in the last 60 minutes

then filter the results through xargs which will do an rsync to the destination

you can use whatever rsync options you want instead of -tgcop as i used in the example

you can also have multiple rsync processes by adding -P x (something less than 8 for performance reasons) right after xargs though you could leave it out if you have very large files

replace /bar/ with the actual local destination

as far as a remote server is concerned, to automate you would need to setup ssh keys to use without prompting for a password

save it as backup.sh and add cron entry

0 * * * * /path/backup.sh

Open in new window

0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 250 total points
ID: 39893014
rsync can handle the above by itself by using --compare-dest

-------

rsync has an option to do incremental backups by creating hardlinks of files that did not change using --link-dest. it needs a little scripting to be actually usable in cron. here is an example script to handle incremental backups. the script assumes the destination dir to be local but you can adapt differently. it is just simpler to pull rather than push in this context

# set these up
src=/local/dir
dst=/other/dir

test -L $dst.latest && lnk="--link-dest=$dst.latest"
dst_=$dst.`date +%Y%m%d%H%M%S`
if rsync -az $lnk $src $dst_.part
then
  mv $dst_.part $dst_
  ln -sfn $dst_ $dst.latest
else
  echo RSYNC FAILURE, removing synchronised files >&2
  rm -rv $dst_.part
  exit 1
fi

# cleanup previously failed/incomplete synchros
rm -frv $dst*.part | sed 's/^/RM:	/'

Open in new window


-----------

you can also use a combination of --backup and --suffix in order to store everything in a single directory. use the current date as suffix for example.
0

Featured Post

Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

By default, Carbonite Server Backup manages your encryption key for you using Advanced Encryption Standard (AES) 128-bit encryption. If you choose to manage your private encryption key, your backups will be encrypted using AES 256-bit encryption.
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question