Upload only modified files to my server

Hello Jonah11,

The tool you need is rsync. If you have ssh access to the remote server.

http://en.wikipedia.org/wiki/Rsync
http://www.samba.org/rsync/ (check examples)

Regards.

Hi,

Could you give more info on what is needed to setup rsync.  I've found a few web tutorials on it but none of them are clear to me.  Will I need to install something on my server as well as my local windows box?

Also, should I consider cwRsync (http://www.itefix.no/i2/node/10650) or DeltaCopy (another GUI wrapper around rsync for windows).  I tried downloading DeltaCopy but it asks me to enter a server IP/Host Name and doesn't have a space for username or password as ftp does.  What protocol is it using?  If I am uploading to a secure directory, I don't see how it can get by without a user/pass.

Thanks,
Jonah

leakim,

thanks. i do have ssh access. but what do you mean by nothing else on my web server?

Sorry for my poor english ^^
No need to install something on the server. Only the ssh access is required.

Regards.

Hi leakim,

Ok, I got cwrsync successfully installed on my machine.  and I have it communicating with my server.  however, it's still not working, and it's slow (the command takes about 45 seconds to execute every time I run it).  If I turn on the verbose option, I can see it "sending an incremental file list" which means that it spiders through every file and folder.  It does this every time it is run.  It does not work in this sense: If I change a file, run cwrsync, and then check that file on the server, it has not been changed.  

I am running the attached batch file, which comes with cwrsync.  Only the final line has been changed by me.

What could be going wrong?

@ECHO OFF
REM *****************************************************************
REM
REM CWRSYNC.CMD - Batch file template to start your rsync command (s).
REM
REM By Tevfik K. (http://itefix.no)
REM *****************************************************************
 
REM Make environment variable changes local to this batch file
SETLOCAL
 
REM ** CUSTOMIZE ** Specify where to find rsync and related files (C:\CWRSYNC)
SET CWRSYNCHOME=%PROGRAMFILES%\CWRSYNC
 
REM Set CYGWIN variable to 'nontsec'. That makes sure that permissions
REM on your windows machine are not updated as a side effect of cygwin
REM operations.
SET CYGWIN=nontsec
 
REM Set HOME variable to your windows home directory. That makes sure 
REM that ssh command creates known_hosts in a directory you have access.
SET HOME=%HOMEDRIVE%%HOMEPATH%
 
REM Make cwRsync home as a part of system PATH to find required DLLs
SET CWOLDPATH=%PATH%
SET PATH=%CWRSYNCHOME%\BIN;%PATH%
 
REM Windows paths may contain a colon (:) as a part of drive designation and 
REM backslashes (example c:\, g:\). However, in rsync syntax, a colon in a 
REM path means searching for a remote host. Solution: use absolute path 'a la unix', 
REM replace backslashes (\) with slashes (/) and put -/cygdrive/- in front of the 
REM drive letter:
REM 
REM Example : C:\WORK\* --> /cygdrive/c/work/*
REM 
REM Example 1 - rsync recursively to a unix server with an openssh server :
REM
REM       rsync -r /cygdrive/c/work/ remotehost:/home/user/work/
REM
REM Example 2 - Local rsync recursively 
REM
REM       rsync -r /cygdrive/c/work/ /cygdrive/d/work/doc/
REM
REM Example 3 - rsync to an rsync server recursively :
REM    (Double colons?? YES!!)
REM
REM       rsync -r /cygdrive/c/doc/ remotehost::module/doc
REM
REM Rsync is a very powerful tool. Please look at documentation for other options. 
REM
 
REM ** CUSTOMIZE ** Enter your rsync command(s) here
REM
REM       ADDED BY JONAH
 
rsync -r /cygdrive/e/data/documents/myproject username@99.999.999.999:~/public_html/myproject
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:

Hello Jonah11,

If you need more speed and are able to know which subfolder you need to update, create multiples scripts for multiples folders.
You can try to sync a little folder with one file to see if its a "bandwidth" problem.

Regards.

What about the fact that its not uploading files when I change them?

Think the inventory is not fully accomplished.
So please try with one folder and a light file to see what happen.

Regards.

Yes,

1 file worked just fine.  Can you explain why it wasn't working in the other case though?  If the program breaks when you have too many files, it seems kind of pointless.....

Not many files but low bandwidth between you and your web server for the first sync and transferts.
If you want to be sure and if you can create a virtual server on your local network to check.

I will review the rsync doc to see if you may add some option for "low" bandwidth.

You may create multiple sync script to sync folder by folder, one for pictures, an other for admin templates, ...

Regards.

hmmm... i still don't understand what you are proposing is happening.  I have 200KB/s upload speed and 1000KB/s downloadspeed, fwiw.  so I don't think low bandwidth is the problem.  and the process is definitely completing.

what criteria does rsync use to decide if it needs to upload a file?  also, is there any way to have it output ONLY the files that have changed and that it is sending?  the verbose option gives way too much information about incremental transfers and whatnot.

Thanks,
Jonah

200KB/s upload ? Wow ! Forget my comments about bandwidth ^^;

Run this to see what files is transfered :

rsync -r -n -v /cygdrive/e/data/documents/myproject/ username@99.999.999.999:~/public_html/myproject

Don't forget to add a slash after projet to the source -> ...ents/myproject/
Add the "n" option to see what is sending successfully

Regards.

Ok, when I do that, it says "sending incremental file list"
Then lists several hundred file names (everything in my project)
Then it says:

sent 174134 bytes  received 17442 bytes  8152.17 bytes/sec
total size is 21654308  speedup is 113.03 (DRY RUN)

And then the process completes.  And still my changed files is not being sent.  Again, maybe it has something to with how rsync determines if a change is to be made?  Any other ideas?

Thanks again.

Hello,

Don't know how rsync determine how file need to be update or not.
http://www.samba.org/rsync/how-rsync-works.html

Perhaps it's a "permissions" problem to replace/delete file that need update when rsync try to replace the file ?
I see rsync sent 174134 bytes.
On the web server could you delete one of your file need to be update to see if rsync upload it successfully.

Regards.

I tested deleting a file: it was not replaced by rsync.  Also, here are the permissions of the file on the server from a "ls -l" command (note that rsync is logging in as jonah):

-rw-r--r-- 1 jonah jonah  7373 Jul 22 23:29 bundle.js

Does that give you any hints?  I will look over the link you attached and see if anything seems relevant.

Ok, so I figured it out.  I needed to leave "myproject" off the end of the receiver path.  it turns out all the files were being sent, but they were being sent to a new folder called "myproject" that existed within the existing "myproject" folder on the server, and since there were so many folders i didn't even notice.  also, using --checksum is a good option for me, since my local dates are all newer than my server dates.

anyway, thanks for all your help!

Excellent work troubleshooting.  Very helpful.