Link to home
Start Free TrialLog in
Avatar of Jonah11
Jonah11

asked on

Upload only modified files to my server

I'm working on a website on a local windows computer, and after making updates I need to upload only the changed files to my web server, which is a linux machine.  What would be the best solution for this?  I want to avoid installing svn, and I want a solution that works quickly, and doesn't have compare all files every time I request an upload.  That is, on the local side, it should keep an index of changed files.
Avatar of Timoros
Timoros
Flag of Greece image

You can try filezilla but you will manually update the files (new or changed )
http://filezilla-project.org/
Hello Jonah11,

The tool you need is rsync. If you have ssh access to the remote server.

http://en.wikipedia.org/wiki/Rsync
http://www.samba.org/rsync/ (check examples)

Regards.
Avatar of Jonah11
Jonah11

ASKER

Hi,

Could you give more info on what is needed to setup rsync.  I've found a few web tutorials on it but none of them are clear to me.  Will I need to install something on my server as well as my local windows box?

Also, should I consider cwRsync (http://www.itefix.no/i2/node/10650) or DeltaCopy (another GUI wrapper around rsync for windows).  I tried downloading DeltaCopy but it asks me to enter a server IP/Host Name and doesn't have a space for username or password as ftp does.  What protocol is it using?  If I am uploading to a secure directory, I don't see how it can get by without a user/pass.

Thanks,
Jonah
ASKER CERTIFIED SOLUTION
Avatar of leakim971
leakim971
Flag of Guadeloupe image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Jonah11

ASKER

leakim,

thanks. i do have ssh access. but what do you mean by nothing else on my web server?
Sorry for my poor english ^^
No need to install something on the server. Only the ssh access is required.

Regards.
Avatar of Jonah11

ASKER

Hi leakim,

Ok, I got cwrsync successfully installed on my machine.  and I have it communicating with my server.  however, it's still not working, and it's slow (the command takes about 45 seconds to execute every time I run it).  If I turn on the verbose option, I can see it "sending an incremental file list" which means that it spiders through every file and folder.  It does this every time it is run.  It does not work in this sense: If I change a file, run cwrsync, and then check that file on the server, it has not been changed.  

I am running the attached batch file, which comes with cwrsync.  Only the final line has been changed by me.

What could be going wrong?
@ECHO OFF
REM *****************************************************************
REM
REM CWRSYNC.CMD - Batch file template to start your rsync command (s).
REM
REM By Tevfik K. (http://itefix.no)
REM *****************************************************************
 
REM Make environment variable changes local to this batch file
SETLOCAL
 
REM ** CUSTOMIZE ** Specify where to find rsync and related files (C:\CWRSYNC)
SET CWRSYNCHOME=%PROGRAMFILES%\CWRSYNC
 
REM Set CYGWIN variable to 'nontsec'. That makes sure that permissions
REM on your windows machine are not updated as a side effect of cygwin
REM operations.
SET CYGWIN=nontsec
 
REM Set HOME variable to your windows home directory. That makes sure 
REM that ssh command creates known_hosts in a directory you have access.
SET HOME=%HOMEDRIVE%%HOMEPATH%
 
REM Make cwRsync home as a part of system PATH to find required DLLs
SET CWOLDPATH=%PATH%
SET PATH=%CWRSYNCHOME%\BIN;%PATH%
 
REM Windows paths may contain a colon (:) as a part of drive designation and 
REM backslashes (example c:\, g:\). However, in rsync syntax, a colon in a 
REM path means searching for a remote host. Solution: use absolute path 'a la unix', 
REM replace backslashes (\) with slashes (/) and put -/cygdrive/- in front of the 
REM drive letter:
REM 
REM Example : C:\WORK\* --> /cygdrive/c/work/*
REM 
REM Example 1 - rsync recursively to a unix server with an openssh server :
REM
REM       rsync -r /cygdrive/c/work/ remotehost:/home/user/work/
REM
REM Example 2 - Local rsync recursively 
REM
REM       rsync -r /cygdrive/c/work/ /cygdrive/d/work/doc/
REM
REM Example 3 - rsync to an rsync server recursively :
REM    (Double colons?? YES!!)
REM
REM       rsync -r /cygdrive/c/doc/ remotehost::module/doc
REM
REM Rsync is a very powerful tool. Please look at documentation for other options. 
REM
 
REM ** CUSTOMIZE ** Enter your rsync command(s) here
REM
REM       ADDED BY JONAH
 
rsync -r /cygdrive/e/data/documents/myproject username@99.999.999.999:~/public_html/myproject

Open in new window

Hello Jonah11,

If you need more speed and are able to know which subfolder you need to update, create multiples scripts for multiples folders.
You can try to sync a little folder with one file to see if its a "bandwidth" problem.

Regards.
Avatar of Jonah11

ASKER

What about the fact that its not uploading files when I change them?
Think the inventory is not fully accomplished.
So please try with one folder and a light file to see what happen.

Regards.
Avatar of Jonah11

ASKER

Yes,

1 file worked just fine.  Can you explain why it wasn't working in the other case though?  If the program breaks when you have too many files, it seems kind of pointless.....
Not many files but low bandwidth between you and your web server for the first sync and transferts.
If you want to be sure and if you can create a virtual server on your local network to check.

I will review the rsync doc to see if you may add some option for "low" bandwidth.

You may create multiple sync script to sync folder by folder, one for pictures, an other for admin templates, ...

Regards.
Avatar of Jonah11

ASKER

hmmm... i still don't understand what you are proposing is happening.  I have 200KB/s upload speed and 1000KB/s downloadspeed, fwiw.  so I don't think low bandwidth is the problem.  and the process is definitely completing.

what criteria does rsync use to decide if it needs to upload a file?  also, is there any way to have it output ONLY the files that have changed and that it is sending?  the verbose option gives way too much information about incremental transfers and whatnot.

Thanks,
Jonah
200KB/s upload ? Wow ! Forget my comments about bandwidth ^^;

Run this to see what files is transfered :

rsync -r -n -v /cygdrive/e/data/documents/myproject/ username@99.999.999.999:~/public_html/myproject

Don't forget to add a slash after projet to the source -> ...ents/myproject/
Add the "n" option to see what is sending successfully

Regards.
Avatar of Jonah11

ASKER

Ok, when I do that, it says "sending incremental file list"
Then lists several hundred file names (everything in my project)
Then it says:

sent 174134 bytes  received 17442 bytes  8152.17 bytes/sec
total size is 21654308  speedup is 113.03 (DRY RUN)

And then the process completes.  And still my changed files is not being sent.  Again, maybe it has something to with how rsync determines if a change is to be made?  Any other ideas?

Thanks again.
Hello,

Don't know how rsync determine how file need to be update or not.
http://www.samba.org/rsync/how-rsync-works.html

Perhaps it's a "permissions" problem to replace/delete file that need update when rsync try to replace the file ?
I see rsync sent 174134 bytes.
On the web server could you delete one of your file need to be update to see if rsync upload it successfully.

Regards.
Avatar of Jonah11

ASKER

I tested deleting a file: it was not replaced by rsync.  Also, here are the permissions of the file on the server from a "ls -l" command (note that rsync is logging in as jonah):

-rw-r--r-- 1 jonah jonah  7373 Jul 22 23:29 bundle.js

Does that give you any hints?  I will look over the link you attached and see if anything seems relevant.

Avatar of Jonah11

ASKER

Ok, so I figured it out.  I needed to leave "myproject" off the end of the receiver path.  it turns out all the files were being sent, but they were being sent to a new folder called "myproject" that existed within the existing "myproject" folder on the server, and since there were so many folders i didn't even notice.  also, using --checksum is a good option for me, since my local dates are all newer than my server dates.

anyway, thanks for all your help!
Avatar of Jonah11

ASKER

Excellent work troubleshooting.  Very helpful.
Congratulations !