Use a .netrc file for the userid/password entry.
Create an input file for the ftp job that looks like:
delete yesterday1
delete yesteday2
etc
mget * (into an empty directory)
quit
Then as you process each input file, create a list of file names for tomorrow's deletes.
To prevent processing partially created files:
Create two files on the remote server, the first contain the data, the second is a semaphore, same file name, different extension, minimal size.
When you process the received files, select only the ones for which a semaphore file exists.
The next day, you will download any files that were partially available the day before, but this time there will be a semaphore file present.
Main Topics
Browse All Topics
by: arnoldPosted on 2009-11-01 at 14:19:23ID: 25715862
you can use wget or ncftp, you might want to consider scp and rsync as an alternative.
You would need to script the retrieve directory listing. Then using the data compare to prior data or use date stamps to requests only new files. If there is a specific size the file must have, you could check whether the listing reflects a "valid" size or use a check to see whether the file is older than 10 minutes ago to be sure that the upload was done prior to retrieving the file.
There are perl/expect scripts that you could also use to connect to the FTP server and then perform the tasks as needed.