?
Solved

Rsync Batch Directory Synchronisation to Remote Server

Posted on 2006-11-01
11
Medium Priority
?
674 Views
Last Modified: 2008-01-09
I have two server's...

1 called will01, the other will02.

Will01 is our primary linux server, which serves our internal intranet, mysql databases, file services and mail service's.

The key directory's I need to sychronise are /home and /vol1.

These need to be schronised to to will02 /home and /vol1.

No files must get deleted from will01, but they should be deleted from will02 if they don't exist on will01.

I think this can be done using rsync, what command would I use? Also as this will run in a cron job, I cannot have rsync asking for a password. This is a farely urgent deadline required for this.
0
Comment
Question by:Terry Rogers
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 3
  • 2
  • +1
11 Comments
 
LVL 14

Expert Comment

by:ygoutham
ID: 17848330
you can do it with rsync or a simple nfs mount and rsync if you do not want to supply passwords.

NFS

in will01 edit the file /etc/exports and add the following lines

/home  ip.address.of.will02(ro)
/vol1   ip.address.of.will02(ro)

close the file and "service nfs start". if already running, then "exportfs -r".

in will02 you can simply mount the directories of will01

mount -t nfs -o soft,timeo=10 ip.address.of.will01:/home  /some/mount/point

and same for /vol1 with /some/other/mount.  now you have them locally mounted for will02 where you can run rsync

rsync -avz  --delete /some/mount/point/   /path/to/backup/directory

the --delete is for removing any file which is deleted in will01 to be automatically deleted from will02.  since it is only a read only share, no file should get deleted from will01.

******************8

you can also copy the file in will02 =>   /root/.ssh/id_rsa.pub and put int will01  =>   /root/.ssh/authorized_keys

this will give you a no password required (depending on your /etc/ssh/sshd_config ) sign in option to will01.  then you run only the rsync from will02 with

rsync -avz  --delete will01:/home   /path/to/backup/directory

however, the second option is riskier since there is no password required kind of option.

0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17848343
I am aware of nfs but would prefer not to use this method.

I have read in a few places that you can set a variable with the password in for rsync or use an external file. This is my preferred method.

Also the script will run on will01 replicating to will02, not running on will02 replicating from will01.
0
 
LVL 14

Expert Comment

by:ygoutham
ID: 17848376
then why not try the /etc/rsync.conf file and have a simple samba like sharing enabled.  then you do not need any password or any other stuff.  you just have to run rsyncd and any machine with a rsync command can copy the files from remote.

edit /etc/rsync.conf


this is mine
**********

uid = nobody
       gid = nobody
       use chroot = no
       max connections = 4
       syslog facility = local5
       pid file = /var/run/rsyncd.pid
                                                                               
       [ftp]
               path = /var/ftp/pub
               comment = whole ftp area (approx 6.1 GB)
                                                                               
      [www]
                path = /var/www/html
                comment = whole php area
      [mysql]
                path = /var/lib/mysql
                comment = mysql databases (approx 1 GB)
*********

you can even add a hosts allow option to specify only your machine ip address so that it is not sharing the same to the rest of the world.

more info on "man rsyncd.conf"

you can specify the password to be used only for rsync in a file /etc/rsyncd.secrets

username:password

i would say that is still not a good idea to store passwords in text files accessible to all and sundry...

*********

:-(  please interchange will01 and will02 in the previous post then that should be the same.  i was try to put across an idea to you. that was all.
0
Get MySQL database support online, now!

At Percona’s web store you can order your MySQL database support needs in minutes. No hassles, no fuss, just pick and click. Pay online with a credit card.

 
LVL 14

Expert Comment

by:ygoutham
ID: 17848385
with this you can just run a rsync like

rsync -avz --delete will01:www  /some/local/path

no need to bother with the path and other areas.  you can also do a chroot = yes so that upward traversing can be disabled if security is a need.
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17848394
Thats is fine, except this smb.conf as well as mail and apache and mysql configuration files also need to be sychronised.

I was under the impression it is quite easy to use rsync for this purpose? The problem I have is the lack of understanding the correct switches to use and getting over the password prompt.

I was using the following command
rsync -HDgoptavz /vol1 root@will02:/vol1
but am uncertain if this will achieve the correct result I am after, plus it prompts for the password.

I tried creating a script file (As there will be several rsync commands I will need to issue) called replicate and put the following in there ...

SET RSYNC_PASSWORD=xxx (Our password)
rsync -HDgoptavz /vol1 root@will02:/vol1

In the hope that this would prevent the password prompt.

So there are two seperate issues here...

1. What switches do I need to replicate files from will01 to will02 without deleting files on will01, but deleteing on will02.
2. How can I run the command in a batch file and not be prompted for the password.
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17848401
Security is not a HUGE concern (Obviously I don't want to open the server up more than I have to) as these are servers on an internal network blocked off from the outside world.
0
 
LVL 23

Accepted Solution

by:
Mysidia earned 800 total points
ID: 17848406
Then to run it on  will01,  use

rsync -av --delete -e ssh   /home   destinationusername@will02:/destination/path/for/home
rsync -av --delete -e ssh   /vol1   destinationusername@will02:/destination/path/for/vol1


To make the script run, you need to have generated a ssh key without a passphrase for the
individual user the script runs as on will01,

And copied the SSH public key it into  /home/desinationtusername/.ssh/authorized_keys   on will02

To generate a SSH key if you have never done so before, you run

ssh-keygen -t dsa

Just press enter without typing anything, when you're prompted to pick a passphrase.
This generates ~currentuser/.ssh/id_dsa.pub   and    ~currentuser/.ssh/id_dsa


id_dsa.pub   is the file that has a line containing your public key  (the id_dsa file is the secret portion).
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17849426
I have followed your instructions, but to no avail. The rsync command works ok, but still requests password. I have copy the generated files to /home/root/.ssh/authorized_keys/ as mentioned above. I had to create the authorized_keys directory.

Is there any configuration on SSHD's config file's I need to check/change?
0
 
LVL 10

Assisted Solution

by:ssvl
ssvl earned 200 total points
ID: 17849485
authorized_keys is a file you have to copy paste the created public key to this file

just like

cat /somedir/id_rsa.pub >> /roots_homedir(which user you want)/.ssh/authorized_keys

IS you lonin as a root or you su to root??

0
 
LVL 10

Expert Comment

by:ssvl
ID: 17849515
use rsa or dsa what you created

after you pasted you will check via ssh to remote host

ssh remote_host

If this is login in remote host with out asking password then only your rsync will work.
0
 
LVL 1

Author Comment

by:Terry Rogers
ID: 17849541
Works now, thanks for your help.
0

Featured Post

Secure Your WordPress Site: 5 Essential Approaches

WordPress is the web's most popular CMS, but its dominance also makes it a target for attackers. Our eBook will show you how to:

Prevent costly exploits of core and plugin vulnerabilities
Repel automated attacks
Lock down your dashboard, secure your code, and protect your users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

rdate is a Linux command and the network time protocol for immediate date and time setup from another machine. The clocks are synchronized by entering rdate with the -s switch (command without switch just checks the time but does not set anything). …
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Suggested Courses
Course of the Month10 days, 12 hours left to enroll

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question