Copy files from server A to B (it's very important for me to fix it fast... please help!)

you can replace the yes| with  -f  option for the rm command !

i suspect that maybe since you are running the script every minute, maybe your tar file is not ready ! and frankly i dont understand how those two lines alone sync your machines !

why dont you use rsync instead, its much easier an cleaner, and you could create a list of exclude for files that are of size 0 !

 /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 > exclude-list.txt

then rsync -arvz /path/locally remote:/remote/path --exclude-from=exclude-list.txt

and instread of doing a cron, put this in an infinite loop ! that starts over whenever first round is finished, something like:

while (true)
do
   //your find command comes here, possibly you can delete stuff here too!
   //you rsync comes here
done

and start this script in the backgroup or at boot time ! you can run it on both machines!

How can i run this in loop?

while (true)
do
  //your find command comes here, possibly you can delete stuff here too!
  //you rsync comes here
done

and can you post here working line please? This is the path
/usr/bin/find /home/bestofbest/public_html/forum/uploads
Servers: 192.168.0.2  192.168.0.3

you can replace the yes| with  -f  option for the rm command !

doesnt wok in centos 5 :(

on 192.168.0.2 , open a file , in this file put:

#!/bin/bash

while (true)
do
   /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 >  /tmp/exclude-list.txt
   rsync -arvz /home/bestofbest/public_html/forum/uploads/ youruser@192.168.0.3::/home/bestofbest/public_html/forum/uploads --exclude-from=/tmp/exclude-list.txt >> transferlog.txt 2>> transferlog.txt

done

then save this file, lets say you call it syncsript.sh
then do chmod +x syncscript.sh
then do ./syncscript.sh &

you can check what is being transferred, you can take a look at the file transferlog.txt ... you can use tail -f transferlog.txt to see what is going on ... live ...

use ctrl + c to stop the tail command ...

cant you replace

  yes | /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 -ok rm {}

with

 /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 -ok rm -f {}

??

[root@server2 cron.d]#  /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 -ok rm -f {}
/usr/bin/find: missing argument to `-ok'




This how I did it-
[root@server1 html]# nano syncsript.sh
#!/bin/bash

while (true)
do
   /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 >  /tmp/exclude-list.txt
   rsync -arvz /home/bestofbest/public_html/forum/uploads/ youruser@192.168.0.3::/home/bestofbest/public_html/forum/uploads --exclude-from=/tmp/exclude-list.txt >> transferlog.txt 2>> transferlog.txt

[root@server1 html]# chmod +x syncsript.sh

[root@server1 html]# ./syncscript.sh &
[1] 25292
[root@server1 html]# -bash: ./syncscript.sh: No such file or directory

[root@server1 html]# ./syncsript.sh &
[1] 26223
[root@server1 html]# ./syncsript.sh: line 8: syntax error: unexpected end of file

ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1296) [sender=2.6.8]
ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1296) [sender=2.6.8]
ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1296) [sender=2.6.8]
ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1296) [sender=2.6.8]
ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1296) [sender=2.6.8]

i forget to put "done" at the end...
now this error appears non stop

ERROR: The remote path must start with a module name not a /
rsync error: error starting client-server protocol (code 5) at main.c(1296) [sender=2.6.8]

ctrl + c not stopping it

If they are on the same network, you could simply NFS mount the directory on server B.  No file transfers needed.

Can you be more specific please? I dont know what it is, and I dont know how to do it. My server connected with USB cable and each has his own internet connection and IP...

The servers A & B connected with USB cable too&

Yes! They are on same network& checking you solution :)

[root@server1 etc]# nano exports
[root@server1 etc]# nano exports
[root@server1 etc]# exportfs -a
exportfs: No options for /home/bestofbest/public_html/forum/uploads 192.168.0.3: suggest 192.168.0.3(sync) to avoid warning


[root@server2 public_html]# mount 192.168.0.2:/home/bestofbest/public_html/forum/uploads /home/bestofbest/public_html/forum/uploads
mount: mount to NFS server '192.168.0.2' failed: RPC Error: Program not registered.

Sorry, the exports entry should be:

/home/bestofbest/public_html/forum/uploads 192.168.0.3(rw)

Change rw to ro, if you don't need read/write access on serverB

[root@server1 etc]# nano exports
[root@server1 etc]# exportfs -a
[root@server1 etc]#

[root@server2 public_html]# mount 192.168.0.2:/home/bestofbest/public_html/forum/1 /home/bestofbest/public_html/forum/1
mount: mount to NFS server '192.168.0.2' failed: RPC Error: Program not registered.

i think you have 2 colons instead of one in the remote part of the rsync command

to stop the job do, in the same console you started your sync in::

jobs

you will see which jobs you have in the background

it will probably have number 1 so do:

%1

this will bring it to the forground

ctrl +c kills it

for testing , start it without the & for now, when it works fine, use the & ...

[root@server1 html]# nano syncsript.sh
[root@server1 html]# ./syncsript.sh
root@192.168.0.3's password:

Btw, it will copy only new files?
I need to copy from server A to B & from B to A

yes, it will copy only new files not that are not in the list ...

run the same script on the other machine changing the destinatin host ip in the rsync command ... the rest should be the same ...

now in order to not ask you questions about the password, search how ssh-keyrings and ssh without password, ssh keys, you can find it on google or here on experts-exchange ...

Sorry, I forgot that you need to start NFS.

On serverA, do

/etc/init.d/nfs start
/etc/init.d/portmap start

on serverB, do

/etc/init.d/portmap start

can't find any thing to remove the password :(




[root@server1 ~]# mount 192.168.0.3:/home/bestofbest/public_html/forum/1 /home/b                               estofbest/public_html/forum/1
mount: mount to NFS server '192.168.0.3' failed: RPC Error: Authentication error                               .
[root@server1 ~]#



which method is better?

https://www.experts-exchange.com/questions/21004731/how-to-ssh-without-a-password.html

http://www.cs.umd.edu/~arun/misc/ssh.html

its so complicated( cant I just write the user and password at the line?

rsync -arvz /home/bestofbest/public_html/forum/uploads/ USER+PASSWORD@192.168.0.3:super2010:/home/bestofbest/public_$
--exclude-from=/tmp/exclude-list.txt >> transferlog.txt 2>> transferlog.txt

i did it
https://www.experts-exchange.com/questions/21004731/how-to-ssh-without-a-password.html

but still password been asked...

yahooooo!!! yesss! it's working :))))))))
thank you guys!

So how I run this in loop at the background always? and automatically at startup?

"you can check what is being transferred, you can take a look at the file transferlog.txt ... you can use tail -f transferlog.txt to see what is going on ... live ..."

as i enter ./syncscript.sh & or ./syncscript.sh
I see it live...

Strangely, when I get out of putty, files stop transfer :(

1)thats exactly why i said to start it in the background using "&"

2)if you dont use the >> and 2>> to redirect output, you will see it on the screen ...

so check if you have used >> and 2>> as instructed above ...

I see it LIVE with "&" and without :(
it's working good... I just need it to start and work by him self.

its like this&
I open putty and enter my two servers and start the script with ./syncsript.sh &
And its working&

But when I close putty, it stops!

do you have this:

>> transferlog.txt 2>> transferlog.txt

at the end of the rsync line ?

yes!

this is exactly what i have:
#!/bin/bash

while (true)
do
   /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 >  /tmp/exclude-list.txt
   rsync -arvz /home/bestofbest/public_html/forum/uploads/ 192.168.0.3:/home/bestofbest/public_html/forum/uploads
--exclude-from=/tmp/exclude-list.txt >> transferlog.txt 2>> transferlog.txt

done

it contains alot of same lines...

./syncsript.sh: line 7: --exclude-from=/tmp/exclude-list.txt: No such file or directory
./syncsript.sh: line 7: --exclude-from=/tmp/exclude-list.txt: No such file or directory
./syncsript.sh: line 7: --exclude-from=/tmp/exclude-list.txt: No such file or directory
./syncsript.sh: line 7: --exclude-from=/tmp/exclude-list.txt: No such file or directory
./syncsript.sh: line 7: --exclude-from=/tmp/exclude-list.txt: No such file or directory
./syncsript.sh: line 7: --exclude-from=/tmp/exclude-list.txt: No such file or directory

inside the while do ... done should be 2 lines ... the find line and the rsync line ... the exclude part should be part of the second line, and not in a new line ... please make sure thats how it is written !

they're there... this is what i have in the script-

#!/bin/bash

while (true)
do
   /usr/bin/find /home/bestofbest/public_html/forum/uploads -size 0 >  /tmp/exclude-list.txt
   rsync -arvz /home/bestofbest/public_html/forum/uploads/ 192.168.0.3:/home/bestofbest/public_html/forum/uploads
--exclude-from=/tmp/exclude-list.txt >> transferlog.txt 2>> transferlog.txt

done

this :

--exclude-from=/tmp/exclude-list.txt >> transferlog.txt 2>> transferlog.txt

must be part of the rsync line ... not a seperate line , the symptoms you are describing show that they are on two lines, if you still cant get it on one line (for some reason i cant figure) put a slash like this -> "\" at the end of the line that has the rsync command!

Now it's not copy files at all!

it the .txt file i have this
sent 64006 bytes  received 20 bytes  128052.00 bytes/sec
total size is 26299783  speedup is 410.77
building file list ... done

sent 64006 bytes  received 20 bytes  128052.00 bytes/sec
total size is 26299783  speedup is 410.77
building file list ... done

sent 64006 bytes  received 20 bytes  128052.00 bytes/sec
total size is 26299783  speedup is 410.77
building file list ... done

etc...

congratulations ! its working !!! it finished comparing ... and decided there is no need to update files, all syncronized :D

You know what ... after all ... i think using rsync is a bad idea ... especially in short periods loop like yours ... i think it would be better to follow tintin advice ... use NFS to unify storage is better than rsync both ways ... rsync back and forth could corrupt your files if they are rsynced while being written in the first place !

No, it doesnt.  I checked it by uploading new files after running the script. The only thing its doing is putting what I have been seen as live at the text file nonstop, but files not transferring at all. The file get also huge 1gb in 1 hour :(

Btw someone here -
https://www.experts-exchange.com/questions/23730436/Problem-with-running-script-in-Centos-5.html?cid=239&anchorAnswerId=22474427#a22474427
said its not a good idea to put it on loop(

Where does it say its not a good idea to put it in a loop ?

check if the file you uploaded is listed in /tmp/exclude-list.txt ... it shouldnt !

The file grows quickly because rsync works constantly, you could discard rsync output and thus not have this file at all ... but then you will not be able to know what files got transferred ... another solution would be (and makes a lot of sense) to put the script to sleep every iteration ... or ofcourse, run the script from crontab ...

to make the script sleep for 180 seconds for example put:

sleep 180

somewhere inside the loop


Again, i think now that using NFS is better in your case

Okay. Can you give me working script with NFS please? So itll working at the background + turning on every time I reboot my mechine.

*i don't need logs.
*i want it to work in loop always.

Well, Regarding NFS, make sure that you have NFS server running on one of the machines, then follow tintin directions ! then add appropriate entries to /etc/fstab to have it always running when you bootup the machine ... this is also described by tintin post .

can't!!

mount 192.168.0.2:/home/bestofbest/public_html/forum/uploads /home/bestofbest/public_html/forum/uploads
mount: mount to NFS server '192.168.0.2' failed: RPC Error: Authentication error.

I have no idea why you are getting authentication error .

try checking http://osdir.com/ml/linux.admin.managers/2003-01/msg00012.html and https://www.experts-exchange.com/questions/20910462/NFS-access-failed-for-server-XXXX-error-7-RPC-Authentication-error.html , maybe its relevent!

this is windows :/
https://www.experts-exchange.com/questions/20910462/NFS-access-failed-for-server-XXXX-error-7-RPC-Authentication-error.html

no solution in the topic...
http://osdir.com/ml/linux.admin.managers/2003-01/msg00012.html

mount: 192.168.0.3:/home/bestofbest/public_html/forum/uploads failed, reason given by server: Permission denied

Thanks alot!

The mount method working just perfect! I just cant understand, how its copy from AtoB and from BtoA when I enable it only on one server AtoB?

Thanks alot!

The mount method working just perfect! I just cant understand, how its copy from AtoB and from BtoA when I enable it only on one server AtoB?

When you NFS mount a directory on another server/s, the two (or more) servers are sharing that disk.  The remote servers treat it like a local disk.

:) just perfect. thaks again...

It was worth the money for pro - user))