urchin
asked on
automating file transfers
I'm trying to set up a cronjob and/or shell script that will connect to an FTP server and download particular files on a regular basis, but I've failed to find any documentation that describes how to do so non-interactively. Does anyone know how to do this?
Thanks,
Pat
Thanks,
Pat
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
You can do this very easily with the "expect" command. It should be in your Linux distribution by default. The expect command is something written that will spawn something, send a string, then expect something in return, send another string, etc.
The manpage is very straightforward on what you can do with it and gives good examples. This is what I use every night for a billing feed where I work:
#!/usr/local/bin/expect
spawn ftp cargill
expect whopgood
send fastconnect\r
expect Password
send fast\!connect\r
expect ftp
send "get load_file\r"
expect ftp
send quit\r
I'll explain each line:
1. The script beginning, what to execute when calling this script
2. The session to spawn
3. Expect the word "whopgood" to show up (who the ftp session starts as)
4. Send the word "fastconnect" as the login. The "\r" is a newline to send.
5. Expect the word "Password" to show up
6. Send the password "fast!connect". The "!" has to be backslashed, and again send a newline with the "\r"
7. Expect the word "ftp" to show up
8. Send the phrase "get load_file" to get my remote file called "load_file".
9. Expect the word "ftp" to show up
10. Send the word "quit" to end the session.
(And no, none of these are valid logins, passwords, or machine names where I work, so don't even try. *grin*)
The manpage is very straightforward on what you can do with it and gives good examples. This is what I use every night for a billing feed where I work:
#!/usr/local/bin/expect
spawn ftp cargill
expect whopgood
send fastconnect\r
expect Password
send fast\!connect\r
expect ftp
send "get load_file\r"
expect ftp
send quit\r
I'll explain each line:
1. The script beginning, what to execute when calling this script
2. The session to spawn
3. Expect the word "whopgood" to show up (who the ftp session starts as)
4. Send the word "fastconnect" as the login. The "\r" is a newline to send.
5. Expect the word "Password" to show up
6. Send the password "fast!connect". The "!" has to be backslashed, and again send a newline with the "\r"
7. Expect the word "ftp" to show up
8. Send the phrase "get load_file" to get my remote file called "load_file".
9. Expect the word "ftp" to show up
10. Send the word "quit" to end the session.
(And no, none of these are valid logins, passwords, or machine names where I work, so don't even try. *grin*)
I'd go with wget... You can even create a file with a list of files to be downloaded. If you're using Demand Dialing, you can just spawn wget from cron at a given time so it reads that file and grabs every file on the list.
ASKER
Thanks to all who provided help on this one! Here's what I found trying to implement your suggestions:
1. the -i option doen't have an effect -- I believe this is used to shut off interactive prompting during multiple file transfers (mget/mput)
2. You need to include the ftp host somewhere, either as
ftp hostname < somefile
or as the first line in somefile itself,
open hostname
3. I still couldn't get the username and password to input non-interactively. I worked around this by using .netrc to auto-login (as described in the ftp man pages). Not ideal in my situation, but hey, it works!
I haven't played around with wget or expect yet, but it's good to know there are other (more robust) options.
Thanks again,
urchin
1. the -i option doen't have an effect -- I believe this is used to shut off interactive prompting during multiple file transfers (mget/mput)
2. You need to include the ftp host somewhere, either as
ftp hostname < somefile
or as the first line in somefile itself,
open hostname
3. I still couldn't get the username and password to input non-interactively. I worked around this by using .netrc to auto-login (as described in the ftp man pages). Not ideal in my situation, but hey, it works!
I haven't played around with wget or expect yet, but it's good to know there are other (more robust) options.
Thanks again,
urchin
ASKER