Jonah11
asked on
Upload only modified files to my server
I'm working on a website on a local windows computer, and after making updates I need to upload only the changed files to my web server, which is a linux machine. What would be the best solution for this? I want to avoid installing svn, and I want a solution that works quickly, and doesn't have compare all files every time I request an upload. That is, on the local side, it should keep an index of changed files.
Hello Jonah11,
The tool you need is rsync. If you have ssh access to the remote server.
http://en.wikipedia.org/wiki/Rsync
http://www.samba.org/rsync/ (check examples)
Regards.
The tool you need is rsync. If you have ssh access to the remote server.
http://en.wikipedia.org/wiki/Rsync
http://www.samba.org/rsync/ (check examples)
Regards.
ASKER
Hi,
Could you give more info on what is needed to setup rsync. I've found a few web tutorials on it but none of them are clear to me. Will I need to install something on my server as well as my local windows box?
Also, should I consider cwRsync (http://www.itefix.no/i2/node/10650) or DeltaCopy (another GUI wrapper around rsync for windows). I tried downloading DeltaCopy but it asks me to enter a server IP/Host Name and doesn't have a space for username or password as ftp does. What protocol is it using? If I am uploading to a secure directory, I don't see how it can get by without a user/pass.
Thanks,
Jonah
Could you give more info on what is needed to setup rsync. I've found a few web tutorials on it but none of them are clear to me. Will I need to install something on my server as well as my local windows box?
Also, should I consider cwRsync (http://www.itefix.no/i2/node/10650) or DeltaCopy (another GUI wrapper around rsync for windows). I tried downloading DeltaCopy but it asks me to enter a server IP/Host Name and doesn't have a space for username or password as ftp does. What protocol is it using? If I am uploading to a secure directory, I don't see how it can get by without a user/pass.
Thanks,
Jonah
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
leakim,
thanks. i do have ssh access. but what do you mean by nothing else on my web server?
thanks. i do have ssh access. but what do you mean by nothing else on my web server?
Sorry for my poor english ^^
No need to install something on the server. Only the ssh access is required.
Regards.
No need to install something on the server. Only the ssh access is required.
Regards.
ASKER
Hi leakim,
Ok, I got cwrsync successfully installed on my machine. and I have it communicating with my server. however, it's still not working, and it's slow (the command takes about 45 seconds to execute every time I run it). If I turn on the verbose option, I can see it "sending an incremental file list" which means that it spiders through every file and folder. It does this every time it is run. It does not work in this sense: If I change a file, run cwrsync, and then check that file on the server, it has not been changed.
I am running the attached batch file, which comes with cwrsync. Only the final line has been changed by me.
What could be going wrong?
Ok, I got cwrsync successfully installed on my machine. and I have it communicating with my server. however, it's still not working, and it's slow (the command takes about 45 seconds to execute every time I run it). If I turn on the verbose option, I can see it "sending an incremental file list" which means that it spiders through every file and folder. It does this every time it is run. It does not work in this sense: If I change a file, run cwrsync, and then check that file on the server, it has not been changed.
I am running the attached batch file, which comes with cwrsync. Only the final line has been changed by me.
What could be going wrong?
@ECHO OFF
REM *****************************************************************
REM
REM CWRSYNC.CMD - Batch file template to start your rsync command (s).
REM
REM By Tevfik K. (http://itefix.no)
REM *****************************************************************
REM Make environment variable changes local to this batch file
SETLOCAL
REM ** CUSTOMIZE ** Specify where to find rsync and related files (C:\CWRSYNC)
SET CWRSYNCHOME=%PROGRAMFILES%\CWRSYNC
REM Set CYGWIN variable to 'nontsec'. That makes sure that permissions
REM on your windows machine are not updated as a side effect of cygwin
REM operations.
SET CYGWIN=nontsec
REM Set HOME variable to your windows home directory. That makes sure
REM that ssh command creates known_hosts in a directory you have access.
SET HOME=%HOMEDRIVE%%HOMEPATH%
REM Make cwRsync home as a part of system PATH to find required DLLs
SET CWOLDPATH=%PATH%
SET PATH=%CWRSYNCHOME%\BIN;%PATH%
REM Windows paths may contain a colon (:) as a part of drive designation and
REM backslashes (example c:\, g:\). However, in rsync syntax, a colon in a
REM path means searching for a remote host. Solution: use absolute path 'a la unix',
REM replace backslashes (\) with slashes (/) and put -/cygdrive/- in front of the
REM drive letter:
REM
REM Example : C:\WORK\* --> /cygdrive/c/work/*
REM
REM Example 1 - rsync recursively to a unix server with an openssh server :
REM
REM rsync -r /cygdrive/c/work/ remotehost:/home/user/work/
REM
REM Example 2 - Local rsync recursively
REM
REM rsync -r /cygdrive/c/work/ /cygdrive/d/work/doc/
REM
REM Example 3 - rsync to an rsync server recursively :
REM (Double colons?? YES!!)
REM
REM rsync -r /cygdrive/c/doc/ remotehost::module/doc
REM
REM Rsync is a very powerful tool. Please look at documentation for other options.
REM
REM ** CUSTOMIZE ** Enter your rsync command(s) here
REM
REM ADDED BY JONAH
rsync -r /cygdrive/e/data/documents/myproject username@99.999.999.999:~/public_html/myproject
Hello Jonah11,
If you need more speed and are able to know which subfolder you need to update, create multiples scripts for multiples folders.
You can try to sync a little folder with one file to see if its a "bandwidth" problem.
Regards.
If you need more speed and are able to know which subfolder you need to update, create multiples scripts for multiples folders.
You can try to sync a little folder with one file to see if its a "bandwidth" problem.
Regards.
ASKER
What about the fact that its not uploading files when I change them?
Think the inventory is not fully accomplished.
So please try with one folder and a light file to see what happen.
Regards.
So please try with one folder and a light file to see what happen.
Regards.
ASKER
Yes,
1 file worked just fine. Can you explain why it wasn't working in the other case though? If the program breaks when you have too many files, it seems kind of pointless.....
1 file worked just fine. Can you explain why it wasn't working in the other case though? If the program breaks when you have too many files, it seems kind of pointless.....
Not many files but low bandwidth between you and your web server for the first sync and transferts.
If you want to be sure and if you can create a virtual server on your local network to check.
I will review the rsync doc to see if you may add some option for "low" bandwidth.
You may create multiple sync script to sync folder by folder, one for pictures, an other for admin templates, ...
Regards.
If you want to be sure and if you can create a virtual server on your local network to check.
I will review the rsync doc to see if you may add some option for "low" bandwidth.
You may create multiple sync script to sync folder by folder, one for pictures, an other for admin templates, ...
Regards.
ASKER
hmmm... i still don't understand what you are proposing is happening. I have 200KB/s upload speed and 1000KB/s downloadspeed, fwiw. so I don't think low bandwidth is the problem. and the process is definitely completing.
what criteria does rsync use to decide if it needs to upload a file? also, is there any way to have it output ONLY the files that have changed and that it is sending? the verbose option gives way too much information about incremental transfers and whatnot.
Thanks,
Jonah
what criteria does rsync use to decide if it needs to upload a file? also, is there any way to have it output ONLY the files that have changed and that it is sending? the verbose option gives way too much information about incremental transfers and whatnot.
Thanks,
Jonah
200KB/s upload ? Wow ! Forget my comments about bandwidth ^^;
Run this to see what files is transfered :
rsync -r -n -v /cygdrive/e/data/documents /myproject / username@99.999.999.999:~/ public_htm l/myprojec t
Don't forget to add a slash after projet to the source -> ...ents/myproject/
Add the "n" option to see what is sending successfully
Regards.
Run this to see what files is transfered :
rsync -r -n -v /cygdrive/e/data/documents
Don't forget to add a slash after projet to the source -> ...ents/myproject/
Add the "n" option to see what is sending successfully
Regards.
ASKER
Ok, when I do that, it says "sending incremental file list"
Then lists several hundred file names (everything in my project)
Then it says:
sent 174134 bytes received 17442 bytes 8152.17 bytes/sec
total size is 21654308 speedup is 113.03 (DRY RUN)
And then the process completes. And still my changed files is not being sent. Again, maybe it has something to with how rsync determines if a change is to be made? Any other ideas?
Thanks again.
Then lists several hundred file names (everything in my project)
Then it says:
sent 174134 bytes received 17442 bytes 8152.17 bytes/sec
total size is 21654308 speedup is 113.03 (DRY RUN)
And then the process completes. And still my changed files is not being sent. Again, maybe it has something to with how rsync determines if a change is to be made? Any other ideas?
Thanks again.
Hello,
Don't know how rsync determine how file need to be update or not.
http://www.samba.org/rsync/how-rsync-works.html
Perhaps it's a "permissions" problem to replace/delete file that need update when rsync try to replace the file ?
I see rsync sent 174134 bytes.
On the web server could you delete one of your file need to be update to see if rsync upload it successfully.
Regards.
Don't know how rsync determine how file need to be update or not.
http://www.samba.org/rsync/how-rsync-works.html
Perhaps it's a "permissions" problem to replace/delete file that need update when rsync try to replace the file ?
I see rsync sent 174134 bytes.
On the web server could you delete one of your file need to be update to see if rsync upload it successfully.
Regards.
ASKER
I tested deleting a file: it was not replaced by rsync. Also, here are the permissions of the file on the server from a "ls -l" command (note that rsync is logging in as jonah):
-rw-r--r-- 1 jonah jonah 7373 Jul 22 23:29 bundle.js
Does that give you any hints? I will look over the link you attached and see if anything seems relevant.
-rw-r--r-- 1 jonah jonah 7373 Jul 22 23:29 bundle.js
Does that give you any hints? I will look over the link you attached and see if anything seems relevant.
ASKER
Ok, so I figured it out. I needed to leave "myproject" off the end of the receiver path. it turns out all the files were being sent, but they were being sent to a new folder called "myproject" that existed within the existing "myproject" folder on the server, and since there were so many folders i didn't even notice. also, using --checksum is a good option for me, since my local dates are all newer than my server dates.
anyway, thanks for all your help!
anyway, thanks for all your help!
ASKER
Excellent work troubleshooting. Very helpful.
Congratulations !
http://filezilla-project.org/