Solved

Rsync Batch Directory Synchronisation to Remote Server

Posted on 2006-11-01
11
640 Views
Last Modified: 2008-01-09
I have two server's...

1 called will01, the other will02.

Will01 is our primary linux server, which serves our internal intranet, mysql databases, file services and mail service's.

The key directory's I need to sychronise are /home and /vol1.

These need to be schronised to to will02 /home and /vol1.

No files must get deleted from will01, but they should be deleted from will02 if they don't exist on will01.

I think this can be done using rsync, what command would I use? Also as this will run in a cron job, I cannot have rsync asking for a password. This is a farely urgent deadline required for this.
0
Comment
Question by:Terry Rogers
  • 5
  • 3
  • 2
  • +1
11 Comments
 
LVL 14

Expert Comment

by:ygoutham
Comment Utility
you can do it with rsync or a simple nfs mount and rsync if you do not want to supply passwords.

NFS

in will01 edit the file /etc/exports and add the following lines

/home  ip.address.of.will02(ro)
/vol1   ip.address.of.will02(ro)

close the file and "service nfs start". if already running, then "exportfs -r".

in will02 you can simply mount the directories of will01

mount -t nfs -o soft,timeo=10 ip.address.of.will01:/home  /some/mount/point

and same for /vol1 with /some/other/mount.  now you have them locally mounted for will02 where you can run rsync

rsync -avz  --delete /some/mount/point/   /path/to/backup/directory

the --delete is for removing any file which is deleted in will01 to be automatically deleted from will02.  since it is only a read only share, no file should get deleted from will01.

******************8

you can also copy the file in will02 =>   /root/.ssh/id_rsa.pub and put int will01  =>   /root/.ssh/authorized_keys

this will give you a no password required (depending on your /etc/ssh/sshd_config ) sign in option to will01.  then you run only the rsync from will02 with

rsync -avz  --delete will01:/home   /path/to/backup/directory

however, the second option is riskier since there is no password required kind of option.

0
 
LVL 1

Author Comment

by:Terry Rogers
Comment Utility
I am aware of nfs but would prefer not to use this method.

I have read in a few places that you can set a variable with the password in for rsync or use an external file. This is my preferred method.

Also the script will run on will01 replicating to will02, not running on will02 replicating from will01.
0
 
LVL 14

Expert Comment

by:ygoutham
Comment Utility
then why not try the /etc/rsync.conf file and have a simple samba like sharing enabled.  then you do not need any password or any other stuff.  you just have to run rsyncd and any machine with a rsync command can copy the files from remote.

edit /etc/rsync.conf


this is mine
**********

uid = nobody
       gid = nobody
       use chroot = no
       max connections = 4
       syslog facility = local5
       pid file = /var/run/rsyncd.pid
                                                                               
       [ftp]
               path = /var/ftp/pub
               comment = whole ftp area (approx 6.1 GB)
                                                                               
      [www]
                path = /var/www/html
                comment = whole php area
      [mysql]
                path = /var/lib/mysql
                comment = mysql databases (approx 1 GB)
*********

you can even add a hosts allow option to specify only your machine ip address so that it is not sharing the same to the rest of the world.

more info on "man rsyncd.conf"

you can specify the password to be used only for rsync in a file /etc/rsyncd.secrets

username:password

i would say that is still not a good idea to store passwords in text files accessible to all and sundry...

*********

:-(  please interchange will01 and will02 in the previous post then that should be the same.  i was try to put across an idea to you. that was all.
0
 
LVL 14

Expert Comment

by:ygoutham
Comment Utility
with this you can just run a rsync like

rsync -avz --delete will01:www  /some/local/path

no need to bother with the path and other areas.  you can also do a chroot = yes so that upward traversing can be disabled if security is a need.
0
 
LVL 1

Author Comment

by:Terry Rogers
Comment Utility
Thats is fine, except this smb.conf as well as mail and apache and mysql configuration files also need to be sychronised.

I was under the impression it is quite easy to use rsync for this purpose? The problem I have is the lack of understanding the correct switches to use and getting over the password prompt.

I was using the following command
rsync -HDgoptavz /vol1 root@will02:/vol1
but am uncertain if this will achieve the correct result I am after, plus it prompts for the password.

I tried creating a script file (As there will be several rsync commands I will need to issue) called replicate and put the following in there ...

SET RSYNC_PASSWORD=xxx (Our password)
rsync -HDgoptavz /vol1 root@will02:/vol1

In the hope that this would prevent the password prompt.

So there are two seperate issues here...

1. What switches do I need to replicate files from will01 to will02 without deleting files on will01, but deleteing on will02.
2. How can I run the command in a batch file and not be prompted for the password.
0
Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

 
LVL 1

Author Comment

by:Terry Rogers
Comment Utility
Security is not a HUGE concern (Obviously I don't want to open the server up more than I have to) as these are servers on an internal network blocked off from the outside world.
0
 
LVL 23

Accepted Solution

by:
Mysidia earned 200 total points
Comment Utility
Then to run it on  will01,  use

rsync -av --delete -e ssh   /home   destinationusername@will02:/destination/path/for/home
rsync -av --delete -e ssh   /vol1   destinationusername@will02:/destination/path/for/vol1


To make the script run, you need to have generated a ssh key without a passphrase for the
individual user the script runs as on will01,

And copied the SSH public key it into  /home/desinationtusername/.ssh/authorized_keys   on will02

To generate a SSH key if you have never done so before, you run

ssh-keygen -t dsa

Just press enter without typing anything, when you're prompted to pick a passphrase.
This generates ~currentuser/.ssh/id_dsa.pub   and    ~currentuser/.ssh/id_dsa


id_dsa.pub   is the file that has a line containing your public key  (the id_dsa file is the secret portion).
0
 
LVL 1

Author Comment

by:Terry Rogers
Comment Utility
I have followed your instructions, but to no avail. The rsync command works ok, but still requests password. I have copy the generated files to /home/root/.ssh/authorized_keys/ as mentioned above. I had to create the authorized_keys directory.

Is there any configuration on SSHD's config file's I need to check/change?
0
 
LVL 10

Assisted Solution

by:ssvl
ssvl earned 50 total points
Comment Utility
authorized_keys is a file you have to copy paste the created public key to this file

just like

cat /somedir/id_rsa.pub >> /roots_homedir(which user you want)/.ssh/authorized_keys

IS you lonin as a root or you su to root??

0
 
LVL 10

Expert Comment

by:ssvl
Comment Utility
use rsa or dsa what you created

after you pasted you will check via ssh to remote host

ssh remote_host

If this is login in remote host with out asking password then only your rsync will work.
0
 
LVL 1

Author Comment

by:Terry Rogers
Comment Utility
Works now, thanks for your help.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Suggested Solutions

This is the error message I got (CODE) Error caused by incompatible libmp3lame 3.98-2 with ffmpeg I've googled this error message and found out sometimes it attaches this note "can be treated with downgrade libmp3lame to version 3.97 or 3.98" …
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now