Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions

Rsync Batch Directory Synchronisation to Remote Server

Posted on 2006-11-01
Last Modified: 2008-01-09
I have two server's...

1 called will01, the other will02.

Will01 is our primary linux server, which serves our internal intranet, mysql databases, file services and mail service's.

The key directory's I need to sychronise are /home and /vol1.

These need to be schronised to to will02 /home and /vol1.

No files must get deleted from will01, but they should be deleted from will02 if they don't exist on will01.

I think this can be done using rsync, what command would I use? Also as this will run in a cron job, I cannot have rsync asking for a password. This is a farely urgent deadline required for this.
Question by:Terry Rogers
  • 5
  • 3
  • 2
  • +1
LVL 14

Expert Comment

ID: 17848330
you can do it with rsync or a simple nfs mount and rsync if you do not want to supply passwords.


in will01 edit the file /etc/exports and add the following lines

/home  ip.address.of.will02(ro)
/vol1   ip.address.of.will02(ro)

close the file and "service nfs start". if already running, then "exportfs -r".

in will02 you can simply mount the directories of will01

mount -t nfs -o soft,timeo=10 ip.address.of.will01:/home  /some/mount/point

and same for /vol1 with /some/other/mount.  now you have them locally mounted for will02 where you can run rsync

rsync -avz  --delete /some/mount/point/   /path/to/backup/directory

the --delete is for removing any file which is deleted in will01 to be automatically deleted from will02.  since it is only a read only share, no file should get deleted from will01.


you can also copy the file in will02 =>   /root/.ssh/id_rsa.pub and put int will01  =>   /root/.ssh/authorized_keys

this will give you a no password required (depending on your /etc/ssh/sshd_config ) sign in option to will01.  then you run only the rsync from will02 with

rsync -avz  --delete will01:/home   /path/to/backup/directory

however, the second option is riskier since there is no password required kind of option.


Author Comment

by:Terry Rogers
ID: 17848343
I am aware of nfs but would prefer not to use this method.

I have read in a few places that you can set a variable with the password in for rsync or use an external file. This is my preferred method.

Also the script will run on will01 replicating to will02, not running on will02 replicating from will01.
LVL 14

Expert Comment

ID: 17848376
then why not try the /etc/rsync.conf file and have a simple samba like sharing enabled.  then you do not need any password or any other stuff.  you just have to run rsyncd and any machine with a rsync command can copy the files from remote.

edit /etc/rsync.conf

this is mine

uid = nobody
       gid = nobody
       use chroot = no
       max connections = 4
       syslog facility = local5
       pid file = /var/run/rsyncd.pid
               path = /var/ftp/pub
               comment = whole ftp area (approx 6.1 GB)
                path = /var/www/html
                comment = whole php area
                path = /var/lib/mysql
                comment = mysql databases (approx 1 GB)

you can even add a hosts allow option to specify only your machine ip address so that it is not sharing the same to the rest of the world.

more info on "man rsyncd.conf"

you can specify the password to be used only for rsync in a file /etc/rsyncd.secrets


i would say that is still not a good idea to store passwords in text files accessible to all and sundry...


:-(  please interchange will01 and will02 in the previous post then that should be the same.  i was try to put across an idea to you. that was all.
Back Up Your Microsoft Windows Server®

Back up all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

LVL 14

Expert Comment

ID: 17848385
with this you can just run a rsync like

rsync -avz --delete will01:www  /some/local/path

no need to bother with the path and other areas.  you can also do a chroot = yes so that upward traversing can be disabled if security is a need.

Author Comment

by:Terry Rogers
ID: 17848394
Thats is fine, except this smb.conf as well as mail and apache and mysql configuration files also need to be sychronised.

I was under the impression it is quite easy to use rsync for this purpose? The problem I have is the lack of understanding the correct switches to use and getting over the password prompt.

I was using the following command
rsync -HDgoptavz /vol1 root@will02:/vol1
but am uncertain if this will achieve the correct result I am after, plus it prompts for the password.

I tried creating a script file (As there will be several rsync commands I will need to issue) called replicate and put the following in there ...

SET RSYNC_PASSWORD=xxx (Our password)
rsync -HDgoptavz /vol1 root@will02:/vol1

In the hope that this would prevent the password prompt.

So there are two seperate issues here...

1. What switches do I need to replicate files from will01 to will02 without deleting files on will01, but deleteing on will02.
2. How can I run the command in a batch file and not be prompted for the password.

Author Comment

by:Terry Rogers
ID: 17848401
Security is not a HUGE concern (Obviously I don't want to open the server up more than I have to) as these are servers on an internal network blocked off from the outside world.
LVL 23

Accepted Solution

Mysidia earned 200 total points
ID: 17848406
Then to run it on  will01,  use

rsync -av --delete -e ssh   /home   destinationusername@will02:/destination/path/for/home
rsync -av --delete -e ssh   /vol1   destinationusername@will02:/destination/path/for/vol1

To make the script run, you need to have generated a ssh key without a passphrase for the
individual user the script runs as on will01,

And copied the SSH public key it into  /home/desinationtusername/.ssh/authorized_keys   on will02

To generate a SSH key if you have never done so before, you run

ssh-keygen -t dsa

Just press enter without typing anything, when you're prompted to pick a passphrase.
This generates ~currentuser/.ssh/id_dsa.pub   and    ~currentuser/.ssh/id_dsa

id_dsa.pub   is the file that has a line containing your public key  (the id_dsa file is the secret portion).

Author Comment

by:Terry Rogers
ID: 17849426
I have followed your instructions, but to no avail. The rsync command works ok, but still requests password. I have copy the generated files to /home/root/.ssh/authorized_keys/ as mentioned above. I had to create the authorized_keys directory.

Is there any configuration on SSHD's config file's I need to check/change?
LVL 10

Assisted Solution

ssvl earned 50 total points
ID: 17849485
authorized_keys is a file you have to copy paste the created public key to this file

just like

cat /somedir/id_rsa.pub >> /roots_homedir(which user you want)/.ssh/authorized_keys

IS you lonin as a root or you su to root??

LVL 10

Expert Comment

ID: 17849515
use rsa or dsa what you created

after you pasted you will check via ssh to remote host

ssh remote_host

If this is login in remote host with out asking password then only your rsync will work.

Author Comment

by:Terry Rogers
ID: 17849541
Works now, thanks for your help.

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question