• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 689
  • Last Modified:

Rsync Batch Directory Synchronisation to Remote Server

I have two server's...

1 called will01, the other will02.

Will01 is our primary linux server, which serves our internal intranet, mysql databases, file services and mail service's.

The key directory's I need to sychronise are /home and /vol1.

These need to be schronised to to will02 /home and /vol1.

No files must get deleted from will01, but they should be deleted from will02 if they don't exist on will01.

I think this can be done using rsync, what command would I use? Also as this will run in a cron job, I cannot have rsync asking for a password. This is a farely urgent deadline required for this.
Terry Rogers
Terry Rogers
  • 5
  • 3
  • 2
  • +1
2 Solutions
you can do it with rsync or a simple nfs mount and rsync if you do not want to supply passwords.


in will01 edit the file /etc/exports and add the following lines

/home  ip.address.of.will02(ro)
/vol1   ip.address.of.will02(ro)

close the file and "service nfs start". if already running, then "exportfs -r".

in will02 you can simply mount the directories of will01

mount -t nfs -o soft,timeo=10 ip.address.of.will01:/home  /some/mount/point

and same for /vol1 with /some/other/mount.  now you have them locally mounted for will02 where you can run rsync

rsync -avz  --delete /some/mount/point/   /path/to/backup/directory

the --delete is for removing any file which is deleted in will01 to be automatically deleted from will02.  since it is only a read only share, no file should get deleted from will01.


you can also copy the file in will02 =>   /root/.ssh/id_rsa.pub and put int will01  =>   /root/.ssh/authorized_keys

this will give you a no password required (depending on your /etc/ssh/sshd_config ) sign in option to will01.  then you run only the rsync from will02 with

rsync -avz  --delete will01:/home   /path/to/backup/directory

however, the second option is riskier since there is no password required kind of option.

Terry RogersIT Senior EngineerAuthor Commented:
I am aware of nfs but would prefer not to use this method.

I have read in a few places that you can set a variable with the password in for rsync or use an external file. This is my preferred method.

Also the script will run on will01 replicating to will02, not running on will02 replicating from will01.
then why not try the /etc/rsync.conf file and have a simple samba like sharing enabled.  then you do not need any password or any other stuff.  you just have to run rsyncd and any machine with a rsync command can copy the files from remote.

edit /etc/rsync.conf

this is mine

uid = nobody
       gid = nobody
       use chroot = no
       max connections = 4
       syslog facility = local5
       pid file = /var/run/rsyncd.pid
               path = /var/ftp/pub
               comment = whole ftp area (approx 6.1 GB)
                path = /var/www/html
                comment = whole php area
                path = /var/lib/mysql
                comment = mysql databases (approx 1 GB)

you can even add a hosts allow option to specify only your machine ip address so that it is not sharing the same to the rest of the world.

more info on "man rsyncd.conf"

you can specify the password to be used only for rsync in a file /etc/rsyncd.secrets


i would say that is still not a good idea to store passwords in text files accessible to all and sundry...


:-(  please interchange will01 and will02 in the previous post then that should be the same.  i was try to put across an idea to you. that was all.
Build your data science skills into a career

Are you ready to take your data science career to the next step, or break into data science? With Springboard’s Data Science Career Track, you’ll master data science topics, have personalized career guidance, weekly calls with a data science expert, and a job guarantee.

with this you can just run a rsync like

rsync -avz --delete will01:www  /some/local/path

no need to bother with the path and other areas.  you can also do a chroot = yes so that upward traversing can be disabled if security is a need.
Terry RogersIT Senior EngineerAuthor Commented:
Thats is fine, except this smb.conf as well as mail and apache and mysql configuration files also need to be sychronised.

I was under the impression it is quite easy to use rsync for this purpose? The problem I have is the lack of understanding the correct switches to use and getting over the password prompt.

I was using the following command
rsync -HDgoptavz /vol1 root@will02:/vol1
but am uncertain if this will achieve the correct result I am after, plus it prompts for the password.

I tried creating a script file (As there will be several rsync commands I will need to issue) called replicate and put the following in there ...

SET RSYNC_PASSWORD=xxx (Our password)
rsync -HDgoptavz /vol1 root@will02:/vol1

In the hope that this would prevent the password prompt.

So there are two seperate issues here...

1. What switches do I need to replicate files from will01 to will02 without deleting files on will01, but deleteing on will02.
2. How can I run the command in a batch file and not be prompted for the password.
Terry RogersIT Senior EngineerAuthor Commented:
Security is not a HUGE concern (Obviously I don't want to open the server up more than I have to) as these are servers on an internal network blocked off from the outside world.
Then to run it on  will01,  use

rsync -av --delete -e ssh   /home   destinationusername@will02:/destination/path/for/home
rsync -av --delete -e ssh   /vol1   destinationusername@will02:/destination/path/for/vol1

To make the script run, you need to have generated a ssh key without a passphrase for the
individual user the script runs as on will01,

And copied the SSH public key it into  /home/desinationtusername/.ssh/authorized_keys   on will02

To generate a SSH key if you have never done so before, you run

ssh-keygen -t dsa

Just press enter without typing anything, when you're prompted to pick a passphrase.
This generates ~currentuser/.ssh/id_dsa.pub   and    ~currentuser/.ssh/id_dsa

id_dsa.pub   is the file that has a line containing your public key  (the id_dsa file is the secret portion).
Terry RogersIT Senior EngineerAuthor Commented:
I have followed your instructions, but to no avail. The rsync command works ok, but still requests password. I have copy the generated files to /home/root/.ssh/authorized_keys/ as mentioned above. I had to create the authorized_keys directory.

Is there any configuration on SSHD's config file's I need to check/change?
authorized_keys is a file you have to copy paste the created public key to this file

just like

cat /somedir/id_rsa.pub >> /roots_homedir(which user you want)/.ssh/authorized_keys

IS you lonin as a root or you su to root??

use rsa or dsa what you created

after you pasted you will check via ssh to remote host

ssh remote_host

If this is login in remote host with out asking password then only your rsync will work.
Terry RogersIT Senior EngineerAuthor Commented:
Works now, thanks for your help.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 5
  • 3
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now