Solved

Rsync Batch Directory Synchronisation to Remote Server

Posted on 2006-11-01
11
650 Views
Last Modified: 2008-01-09
I have two server's...

1 called will01, the other will02.

Will01 is our primary linux server, which serves our internal intranet, mysql databases, file services and mail service's.

The key directory's I need to sychronise are /home and /vol1.

These need to be schronised to to will02 /home and /vol1.

No files must get deleted from will01, but they should be deleted from will02 if they don't exist on will01.

I think this can be done using rsync, what command would I use? Also as this will run in a cron job, I cannot have rsync asking for a password. This is a farely urgent deadline required for this.
0
Comment
Question by:Terry Rogers
  • 5
  • 3
  • 2
  • +1
11 Comments
 
LVL 14

Expert Comment

by:ygoutham
ID: 17848330
you can do it with rsync or a simple nfs mount and rsync if you do not want to supply passwords.

NFS

in will01 edit the file /etc/exports and add the following lines

/home  ip.address.of.will02(ro)
/vol1   ip.address.of.will02(ro)

close the file and "service nfs start". if already running, then "exportfs -r".

in will02 you can simply mount the directories of will01

mount -t nfs -o soft,timeo=10 ip.address.of.will01:/home  /some/mount/point

and same for /vol1 with /some/other/mount.  now you have them locally mounted for will02 where you can run rsync

rsync -avz  --delete /some/mount/point/   /path/to/backup/directory

the --delete is for removing any file which is deleted in will01 to be automatically deleted from will02.  since it is only a read only share, no file should get deleted from will01.

******************8

you can also copy the file in will02 =>   /root/.ssh/id_rsa.pub and put int will01  =>   /root/.ssh/authorized_keys

this will give you a no password required (depending on your /etc/ssh/sshd_config ) sign in option to will01.  then you run only the rsync from will02 with

rsync -avz  --delete will01:/home   /path/to/backup/directory

however, the second option is riskier since there is no password required kind of option.

0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17848343
I am aware of nfs but would prefer not to use this method.

I have read in a few places that you can set a variable with the password in for rsync or use an external file. This is my preferred method.

Also the script will run on will01 replicating to will02, not running on will02 replicating from will01.
0
 
LVL 14

Expert Comment

by:ygoutham
ID: 17848376
then why not try the /etc/rsync.conf file and have a simple samba like sharing enabled.  then you do not need any password or any other stuff.  you just have to run rsyncd and any machine with a rsync command can copy the files from remote.

edit /etc/rsync.conf


this is mine
**********

uid = nobody
       gid = nobody
       use chroot = no
       max connections = 4
       syslog facility = local5
       pid file = /var/run/rsyncd.pid
                                                                               
       [ftp]
               path = /var/ftp/pub
               comment = whole ftp area (approx 6.1 GB)
                                                                               
      [www]
                path = /var/www/html
                comment = whole php area
      [mysql]
                path = /var/lib/mysql
                comment = mysql databases (approx 1 GB)
*********

you can even add a hosts allow option to specify only your machine ip address so that it is not sharing the same to the rest of the world.

more info on "man rsyncd.conf"

you can specify the password to be used only for rsync in a file /etc/rsyncd.secrets

username:password

i would say that is still not a good idea to store passwords in text files accessible to all and sundry...

*********

:-(  please interchange will01 and will02 in the previous post then that should be the same.  i was try to put across an idea to you. that was all.
0
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

 
LVL 14

Expert Comment

by:ygoutham
ID: 17848385
with this you can just run a rsync like

rsync -avz --delete will01:www  /some/local/path

no need to bother with the path and other areas.  you can also do a chroot = yes so that upward traversing can be disabled if security is a need.
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17848394
Thats is fine, except this smb.conf as well as mail and apache and mysql configuration files also need to be sychronised.

I was under the impression it is quite easy to use rsync for this purpose? The problem I have is the lack of understanding the correct switches to use and getting over the password prompt.

I was using the following command
rsync -HDgoptavz /vol1 root@will02:/vol1
but am uncertain if this will achieve the correct result I am after, plus it prompts for the password.

I tried creating a script file (As there will be several rsync commands I will need to issue) called replicate and put the following in there ...

SET RSYNC_PASSWORD=xxx (Our password)
rsync -HDgoptavz /vol1 root@will02:/vol1

In the hope that this would prevent the password prompt.

So there are two seperate issues here...

1. What switches do I need to replicate files from will01 to will02 without deleting files on will01, but deleteing on will02.
2. How can I run the command in a batch file and not be prompted for the password.
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17848401
Security is not a HUGE concern (Obviously I don't want to open the server up more than I have to) as these are servers on an internal network blocked off from the outside world.
0
 
LVL 23

Accepted Solution

by:
Mysidia earned 200 total points
ID: 17848406
Then to run it on  will01,  use

rsync -av --delete -e ssh   /home   destinationusername@will02:/destination/path/for/home
rsync -av --delete -e ssh   /vol1   destinationusername@will02:/destination/path/for/vol1


To make the script run, you need to have generated a ssh key without a passphrase for the
individual user the script runs as on will01,

And copied the SSH public key it into  /home/desinationtusername/.ssh/authorized_keys   on will02

To generate a SSH key if you have never done so before, you run

ssh-keygen -t dsa

Just press enter without typing anything, when you're prompted to pick a passphrase.
This generates ~currentuser/.ssh/id_dsa.pub   and    ~currentuser/.ssh/id_dsa


id_dsa.pub   is the file that has a line containing your public key  (the id_dsa file is the secret portion).
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17849426
I have followed your instructions, but to no avail. The rsync command works ok, but still requests password. I have copy the generated files to /home/root/.ssh/authorized_keys/ as mentioned above. I had to create the authorized_keys directory.

Is there any configuration on SSHD's config file's I need to check/change?
0
 
LVL 10

Assisted Solution

by:ssvl
ssvl earned 50 total points
ID: 17849485
authorized_keys is a file you have to copy paste the created public key to this file

just like

cat /somedir/id_rsa.pub >> /roots_homedir(which user you want)/.ssh/authorized_keys

IS you lonin as a root or you su to root??

0
 
LVL 10

Expert Comment

by:ssvl
ID: 17849515
use rsa or dsa what you created

after you pasted you will check via ssh to remote host

ssh remote_host

If this is login in remote host with out asking password then only your rsync will work.
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17849541
Works now, thanks for your help.
0

Featured Post

Ransomware: The New Cyber Threat & How to Stop It

This infographic explains ransomware, type of malware that blocks access to your files or your systems and holds them hostage until a ransom is paid. It also examines the different types of ransomware and explains what you can do to thwart this sinister online threat.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question