Solved

How to automate a ftp session in my cron job ?

Posted on 2000-04-24
14
646 Views
Last Modified: 2013-12-06
Anybody knows how to automate ftp session in my cron job?  I need to
ftp some files from UNIX server to a PC or NT server on the daily basis.
0
Comment
Question by:shiqi
14 Comments
 
LVL 2

Expert Comment

by:jonke
Comment Utility
You need to use a .netrc file.

With automated  ftp, you set up a ~/.netrc file. In this file are all the login instructions and commands that will be performed.

This file must necessarily have the permissions set to 700 to work, so only the owner can read/write/execute it.

The contents of the .netrc file should look like this:

machine <hostname> login <user_name>  password <the_password>
macdef init
<The ftp instructions e.g.>
put file
get file2
bye



Then to automate the ftp, the command you put in the crontab entry is '/usr/bin/ftp hostname'
Simple as that........

I hope ;-)
0
 
LVL 2

Accepted Solution

by:
jonke earned 30 total points
Comment Utility
Oh- it seems obvious but remember to put it in the crontab of the user who has the file in $HOME.

e.g. if it is the root crontab, then the .netrc file will be in /.netrc
0
 

Author Comment

by:shiqi
Comment Utility
Sorry to response so late due to some reasons.
I have tried your recommendations and I can transfer a file successfully but I still have two more questions.
1. If I want to transfer a file with a variable, how can that be done?  I cannot just put the following
line in .netrc, right?
put myfile.$(date)
2.  If I need to auto-transfer files to two servers everyday, how should I set .netrc?

Thanks.
0
 

Author Comment

by:shiqi
Comment Utility
How to implement  anonymous ftp which can protect files from being overwritten accidently?
0
 
LVL 2

Expert Comment

by:jonke
Comment Utility
PART1

No, I do not think you can put variables in a .netrc  file. The way I have got roung this in the past, is to right a wraparound script for .netrc, and run this from cron. Obviously you'll have to tune this to work with your particular setup e.g.  work out a way of introducing the file variable into the script. I cannot do this as I do not know the file naming conventions that you are using. If you tell me, then I can do this as well though.

So:

#!/bin/ksh
#
#Script to create a .netrc file and automate ftp
#<list of machines> is a file with the list of
#machines you widh to ftp to.
#
for HOSTNAME in `cat <list of machines>`
do
rm ~/.netrc
echo "machine $HOSTNAME login <user_name> password <the_password>" > ~/.netrc
echo "macdef init" >> ~/.netrc
echo "put myfile.$(date)" >> ~/.netrc
bye >> ~/.netrc
#
#Now the .netrc file has been created, we use
#
chmod 700 ~/.netrc
ftp $HOSTNAME
#
#End loop and do the same for the next machine in <list of machines>
#
done
0
 
LVL 2

Expert Comment

by:jonke
Comment Utility
PART2

Doc on how to setup anon ftp:

How to Set Up Anonymous FTP sub-logins for Solaris 2 to restrict user access
(ftp-restricted shell).

NOTE: There is only a slight difference in setting up sub-logins in 2.6
than in 2.5 or 2.5.1.  The security libraries needed have been changed.

Sub-logins for ftp work under Solaris 2.x ONLY!

Anonymous FTP sub-logins:

- Anonymous ftp sub-logins allow you to configure an anonymous FTP
  server, and customize it for different users.

- You can give each user their own home directory and set permissions
  on each home directory so that others can't see or transfer files
  from within them.

- Sub-logins will prevent all users from going beyond the directory
  hierarchy of /export/ftp because anonymous FTP uses the chroot function.

- The user is required to log in twice: once as anonymous and again as
  themselves using the "user" command from the FTP prompt.
  Sublogin takes place when user first logins as anonymous, and then
  issues "user xxx" command - in order to become "xxx", where
  "xxx is a valid username in the anonymous-ftp (chrooted) area
  (ie an entry for "xxx" exists in ~ftp/etc/passwd

- Important Note: The sublogin feature does not work unless nscd caching for
  passwd is  disabled in /etc/nscd.conf. (2.5 or above, see in.ftpd man page)

An example of an Anonymous FTP sub-login:

% ftp ftp.company.com
Connected to ftp.company.com.
220 ftp.company.com FTP server (UNIX(r) System V Release 4.0) ready.
Name (ftp.company.com:joeuser): anonymous
331 Guest login ok, send ident as password.
Password:
230 Guest login ok, access restrictions apply.
ftp> user precious
331 Password required for precious.
Password:
230 User precious logged in.
ftp> pwd
257 "/precious" is current directory.
ftp> ls
200 PORT command successful.
150 ASCII data connection for /bin/ls (129.200.9,41058) (0 bytes).
226 ASCII Transfer complete.
ftp>

----------------------------------------
To Set Up Sub-logins with anonymous FTP:

1. Set up anonymous FTP as you normally would. (see ftpd man page.)

   Note: For the following examples, it is assumed that anonymous ftp is
   set up in the /export/ftp directory.

2. Add the sub-login user entry to the local /etc/passwd file
   create user password - add shadow entry.

   Note: This is necessary for Solaris 2.5, 2.5.1 and 2.6 because the
   in.ftpd daemon gets the home directory, uid, and shell from the
   /etc/passwd or passwd database.

3. Add the user to /export/ftp/etc/passwd file.
   (copied from /etc/passwd)
   example:
        precious:x:30001:30001:ftp sub-login user:/precious:/bin/false

   Note: the home directory is listed as /precious and the shell is
   /bin/false(a bogus shell name). If you are using /bin/false as a shell,
   remember that it needs to be listed in /export/ftp/etc/shells.

4. Create the home directory for the user in /export/ftp with appropriate
   permissions.

   Note: Keep in mind that even though the home directory will be in the
   /export/ftp directory, as far as ftp knows, /export/ftp is really '/'.
   Therefore, the home directory should be listed as /precious in the
   /export/ftp/etc/passwd file, but it will really live at
   /export/ftp/precious.

   Example:

        mkdir /export/ftp/precious
        chown precious /export/ftp/precious
        chmod 755 /export/ftp/precious
 
5. Create/Edit the /export/ftp/etc/shadow file and add the shadow entry with
   the encrypted password( can copy from /etc/shadow file).

   Note: If you are running 2.3 or greater, you MUST have the encrypted passwd
   in the shadow file instead of the "x" in the password field.
   Example of /export/ftp/etc/shadow entry
     precious:LrxyzVVab2c:9561:::::
 
6. Change the permissions on the ftp/shadow file to 400 and change the owner
   on the file to root if it isn't already.

   Example:

        chmod 400 /export/ftp/etc/shadow
        chown root /export/ftp/etc/shadow
 
7. Create a /export/ftp/etc/shells file and add the shell specified in the
   /export/ftp/etc/passwd file.

   Example:
        /bin/false
 
   Note: Use a non-standard shell so that the user will not be able to log in
   as a regular user

This step is unnecessary because this is done for you when the ftp script is
run.
8. Copy the following security libraries: ( for 2.5.1 and 2.6)
    Note : earlier releases of 2.6 had a change in the pam library
    pam_unix.so.1 is now used instead of pam_entry.so.1

        cd /export/ftp
        mkdir usr/lib/security
        (for 2.5.1 and 2.6 Beta)
        cp /usr/lib/security/pam_entry.so.1 usr/lib/security
        ln -s usr/lib/security/pam_entry.so.1 usr/lib/security/pam_entry.so
        (for 2.6 and above )
        cp /usr/lib/security/pam_unix.so.1 usr/lib/security/pam_unix.so.1
        ln -s  usr/lib/security/pam_unix.so.1 usr/lib/security/pam_unix.so

9. cp /etc/pam.conf /export/ftp/etc/pam.conf ( for 2.6 and above )
   
     or create /export/ftp/etc/pam.conf with minimal entries:
          # minimal pam.conf
          ftp   auth      required   /usr/lib/security/pam_unix.so.1
          ftp   account   required   /usr/lib/security/pam_unix.so.1
          ftp   session   required   /usr/lib/security/pam_unix.so.1

10. Modify /etc/nscd.conf to disable passwd caching.(all releases)
    Add the following entry to nscd.conf :
   
        enable-cache            passwd          no
   
    Then restart nscd.
     # nscd -K          
     # nscd
    This can also be temporarily accomplsihed from the command line.
     # nscd -e passwd,"no"

----------------------------------------

Note: to further restrict sub-login users access, private subdirectories can be created,
along with minimum entries in the anonymous ftp shadow , passwd , and group

Example: 2 restricted users ( rhod, and precious ) who do not have access
to each others home dir.

#/etc/passwd
....
ftp:x:30000:30000:Anonymous FTP:/export/ftp:/nosuchshell
precious:x:30001:30001:ftp sub-login user:/precious:/bin/false
rhod:x:9999:1000:hod restricted user:/private/rhod:/bin/false

#/etc/shadow
....
precious:MfLP7b9uFBgvQ:10414::::::
rhod:kIwg.Rrtv.rqU:10325::::::

#/export/ftp/etc/shadow
precious:MfLP7b9uFBgvQ:10414::::::
rhod:kIwg.Rrtv.rqU:10325::::::

#/export/ftp/etc/passwd
root:x:0:1:Super-User:/:/sbin/sh
bin:x:2:2::/usr/bin:
ftp:x:30000:30000:Anonymous FTP:/export/ftp:/nosuchshell
rhod:x:9999:1000:ftp sub-login rhod:/private/rhod:/bin/false
precious:x:30001:30001:ftp sub-login precious:/private/precious:/bin/false

#/export/ftp/etc/shells
/bin/false

#/export/ftp/etc/group
root::0:root
bin::2:root,bin
other::1:
ftp::3000:
subgrp1::1000:rhod
subgrp2::30001:precious

Directory permissions can be set to restrict world access
and the readability of /private area.

# ls -al /export/ftp/private
total 8
drwxr-x--x   4 root     other        512 Mar 16 09:08 .
dr-xr-xr-x   7 root     other        512 Mar 16 09:04 ..
drwxr-x---   2 precious subgrp2      512 Mar 16 09:08 precious
drwxr-x---   3 rhod     subgrp1      512 Mar 16 09:04 rhod
0
 

Author Comment

by:shiqi
Comment Utility
Sorry to reply late.
Thank you very much for your advices.
Now I managed to auto-ftp a daily based file by writing a script similar to yours.  But I really feel that it is not safe to put password in a ASCII file such as .netrc.

As for anonymous FTP,  now I can login as an anonymous user and transfer files.  But now the problem is I cannot run "ls" or "dir" to list the current directory contents after login.   The output is shown as below,
C:\WIN95>ftp nms
Connected to nms.
220 nms FTP server (Version 1.7.212.2 Tue Apr 21 12:14:46 GMT
User (nms:(none)): anonymous
331 Guest login ok, send ident as password.
Password:
230 Guest login ok, access restrictions apply.
ftp> user subftp
331 Password required for subftp.
Password:
230 User subftp logged in.
ftp> dir
200 PORT command successful.
150 Opening ASCII mode data connection for /usr/bin/ls.
226 Transfer complete.
ftp> ls
200 PORT command successful.
150 Opening ASCII mode data connection for file list.
f1
f2
t1
226 Transfer complete.
12 bytes received in 0.00 seconds (12000.00 Kbytes/sec)
ftp> get f1
200 PORT command successful.
150 Opening ASCII mode data connection for f1 (493 bytes).
226 Transfer complete.
502 bytes received in 0.00 seconds (502000.00 Kbytes/sec)
ftp>

 Do you know what could be the reason for this?
 I am using HP-UX 10.20 and it is not a Trusted system.   As you see, "ls" can work but "dir" cannot.   Also,   how come I cannot see how big is the file to be transfered ?     WS_FTP also cannot see anything in the directory after login.

    Another thing that I am wondering is about sub-login,  as mentioned in "man ftpd",  FTP  sub-login IDs do not need to be present in the   system /etc/passwd file. If that is the case,  how can I set or change the passwd?   How can other ftp software like WS_FTP make use of this feature?  It seems quite troublesome that we have to login twice(1. anonymous login; 2. FTP sub-login )
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 2

Expert Comment

by:jonke
Comment Utility
>But I really feel that it is not safe to put >password in a ASCII file such as .netrc.

This is not a security risk. This is as safe as the user account is itself. That is why it will only work if you chmod 700. This means only root and the user can read the file.

>how come I cannot see how big is the file to be >transfered

Use ls -al to see all the files and how big they are.

>sub-login IDs do not need to be present in the   >system /etc/passwd file. If that is the case,  how >can I set or change the passwd?

See the second section, of my last post, regarding sub-logins. Create an entry in the passwd file and shadow file. Then kepp the encrypted passwd entry in the shadow file blank.
0
 

Author Comment

by:shiqi
Comment Utility
But under ftp> prompt I cannot run "ls -l ",  right?  I use WS_FTP utility, also cannot see what is inside the current directory, Why?
0
 
LVL 2

Expert Comment

by:jonke
Comment Utility
The only thing I can suggest here is to use a more up-to-date ftp package. Look for a freeware version on the internet. For example, the ftp package that gets shipped with Solaris 2.x allows ls -al for example.
0
 

Author Comment

by:shiqi
Comment Utility
What I mean is that we often run ftp from PC to UNIX rather than the other way around.  Because we cannot transfer files to PC unless the particular PC is set as a FTP server.  So the ftp package should be on PC side.

If I just use normal user id to run ftp on my PC,  
under "ftp>" I can run either "dir" or "ls -l" to list contents of directories.  But when I use anonymous ftp to do so,  I got nothing, why?
If I can't see what are inside the directory, how do
 I know what files are there for me to transfer?

0
 
LVL 2

Expert Comment

by:jonke
Comment Utility
To be honest, I am not sure if this is a side effect of the pc ftp package you are using, or the anon ftp setup you have on the unix machine. It might be worth trying to reinstall the anon ftp setup on the unix machine. I have just tried the following, and have no problems doing an ls -al, and seeing all the files and directory sizes. If this does not work, then it is a problem you are having with the pc side- the ftp package you are using, and will be something I cannot help you with- as I know next to nothing about pc's.

1)Remove all the anon ftp stuff on the unix server. We might as well start this from scratch. I have a script that will make the installation much easier.

2)Create the anon ftp user in /etc/passwd. This is done by adding the following line:

ftp:x:30000:30000:Anonymous_FTP:/export/ftp:/nosuchshell

3)Add the following line to /etc/shadow:

ftp:NP:6445::::::

4)Save the following to in a vi editor. Save this and chmod it (chmod 744 scriptname), then run it

#!/bin/sh
     # script to setup anonymous ftp area
     #
     # handle the optional command line argument
     case $# in

        # the default location for the anon ftp comes from the passwd file
        0) ftphome="`grep '^ftp:' /etc/passwd | cut -d: -f6`"
           ;;

        1) if [ "$1" = "start" ]; then
              ftphome="`grep '^ftp:' /etc/passwd | cut -d: -f6`"
           else
              ftphome=$1
           fi
           ;;

        *) echo "Usage: $0 [anon-ftp-root]"
           exit 1
           ;;
     esac

     if [ -z "${ftphome}" ]; then
        echo "$0: ftphome must be non-null"
        exit 2
     fi

     # This script assumes that ftphome is neither / nor /usr so ...
     if [ "${ftphome}" = "/" -o "${ftphome}" = "/usr" ]; then
        echo "$0: ftphome must not be / or /usr"
        exit 2
     fi

     # If ftphome does not exist but parent does, create ftphome
     if [ ! -d ${ftphome} ]; then
         # lack of -p below is intentional
         mkdir ${ftphome}
     fi
     echo Setting up anonymous ftp area ${ftphome}

     # Ensure that the /usr/bin directory exists
     if [ ! -d ${ftphome}/usr/bin ]; then
         mkdir -p ${ftphome}/usr/bin
     fi

     cp /usr/bin/ls ${ftphome}/usr/bin
     chmod 111 ${ftphome}/usr/bin/ls

     # Now set the ownership and modes to match the man page
     chown root ${ftphome}/usr/bin
     chmod 555 ${ftphome}/usr/bin

     # this may not be the right thing to do
     # but we need the bin -> usr/bin link
     if [ -r ${ftphome}/bin ]; then
         mv -f ${ftphome}/bin ${ftphome}/Obin
     fi
     ln -s usr/bin ${ftphome}

     # Ensure that the /usr/lib and /etc directories exist
     if [ ! -d ${ftphome}/usr/lib ]; then
         mkdir -p ${ftphome}/usr/lib
     fi
     if [ ! -d ${ftphome}/etc ]; then
         mkdir -p ${ftphome}/etc
     fi

     #Most of the following are needed for basic operation, except
     #for libnsl.so, nss_nis.so, libsocket.so, and straddr.so which are
     #needed to resolve NIS names.

     cp /usr/lib/ld.so /usr/lib/ld.so.1 ${ftphome}/usr/lib

     for lib in libc libdl libintl libw libnsl libsocket         nss_nis nss_nisplus nss_dns nss_files
     do
        cp /usr/lib/${lib}.so.1 ${ftphome}/usr/lib
        rm -f ${ftphome}/usr/lib/${lib}.so
        ln -s ./${lib}.so.1 ${ftphome}/usr/lib/${lib}.so
     done

     cp /usr/lib/straddr.so.2 ${ftphome}/usr/lib
     rm -f ${ftphome}/usr/lib/straddr.so
     ln -s ./straddr.so.2 ${ftphome}/usr/lib/straddr.so

     cp /etc/passwd /etc/group /etc/netconfig ${ftphome}/etc

     # Copy timezone database
     mkdir -p ${ftphome}/usr/share/lib/zoneinfo
     (cd ${ftphome}/usr/share/lib/zoneinfo
       (cd /usr/share/lib/zoneinfo; find . -print | cpio -o) | cpio -imdu
       find . -print | xargs chmod 555
       find . -print | xargs chown root
     )

     chmod 555 ${ftphome}/usr/lib/*
     chmod 444 ${ftphome}/etc/*

     # Now set the ownership and modes
     chown root ${ftphome}/usr/lib ${ftphome}/etc
     chmod 555 ${ftphome}/usr/lib ${ftphome}/etc

     # Ensure that the /dev directory exists
     if [ ! -d ${ftphome}/dev ]; then
         mkdir -p ${ftphome}/dev
     fi

     # make device nodes. ticotsord and udp are necessary for
     # 'ls' to resolve NIS names.

     for device in zero tcp udp ticotsord
     do
        line=`ls -lL /dev/${device} | sed -e 's/,//'`
        major=`echo $line | awk '{print $5}'`
        minor=`echo $line | awk '{print $6}'`
        rm -f ${ftphome}/dev/${device}
        mknod ${ftphome}/dev/${device} c ${major} ${minor}
     done

     chmod 666 ${ftphome}/dev/*

     ## Now set the ownership and modes
     chown root ${ftphome}/dev
     chmod 555 ${ftphome}/dev

     if [ ! -d ${ftphome}/pub ]; then
        mkdir -p ${ftphome}/pub
     fi
     chown ftp ${ftphome}/pub
     chmod 777 ${ftphome}/pub



5)You should now be able to to an anonymous ftp in and read all the files. If not, it is the pc ftp side.
0
 

Expert Comment

by:benson031397
Comment Utility
using .netrc is correct.

But I remember I saw a case that don't need to setup .netrc.

Just write a script, the script's content is similar to the content of .netrc.

Do anyone have any idea?


Benson
0
 
LVL 1

Expert Comment

by:milkmon123
Comment Utility
The netrc class parses and encapsulates the netrc file format used by the Unix ftp program and other FTP clients, so it may be necessary to redo this with another configuration such as...

Return a 3-tuple (login, account, password) of authenticators for host. If the netrc file did not contain an entry for the given host, return the tuple associated with the `default' entry. If neither matching host nor default entry is available, return None....

.. then...

Dump the class data as a string in the format of a netrc file. (This discards comments and may reorder the entries.)

Hope this helps

Matt
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Introduction Regular patching is part of a system administrator's tasks. However, many patches require that the system be in single-user mode before they can be installed. A cluster patch in particular can take quite a while to apply if the machine…
Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now