We help IT Professionals succeed at work.

.netrc works with FTP in an interactive mode but will not run during batch

RSchnabel
RSchnabel asked
on
I have a nightly job that runs on our webservers that uses FTP to download a build-file to our webserver.  

In the cron job the command is issued as follows (where levitcus is the name of the server):
echo "Beginning File Transfer"
/usr/bin/ftp levitcus

The .netrc file contains the following script:
machine levitcus
login XXXXXX
password XXXXXX
macdef init
prompt off
lcd /usr/local/CI-live/CCI/postgresql/data/share/
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/postgresql/data/share/
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit

When I pull the Logs from mail on the webserver I have this line when the FTP Command should be executed:
Login failed.

Thats it thats all I have no extra errors or anything.  When I manually run the command: ftp levitcus everything runs beautifully.

I need help with this as I am really tired of manually triggering the ftp download from home in the wee hours of the morning.

Thanks
Derek
Comment
Watch Question

Pétur Ingi EgilssonSoftware Engineer -- Consultant

Commented:
You can use wget instead of ftp to fetch from FTP's.

wget --ftp-user=XXXX --ftp-password=XXXX ftp://SERVERNAME/FILENAME

Author

Commented:
I have tried working with both wget and curl to download the file.  The issue I have run into both times is that when the file is being downloaded from our AS400 (which builds this file and hosts the file for download) wget and curl were not able to navigate the FTP structure from the AS400.  

I may be doing something wrong but the command I was using was this

wget ftp://USERNAME:PASSWORD@levitcus/astsfdta/SF400X.SF400X --passive-ftp --no-remove-listing -d

The file member name is SF400X in the object SF400X in the library ASTSFDTA.  When wget processes the file the output is text file filled with @ and doesn't seem to ever find the end of the file (well until the HD was filled, I tried this and walked away, didn't work very well for me).  curl gave me the exact same output problem.
Top Expert 2005

Commented:
How have You configured the nightly job? From cron? Sure the user cron entry is run as is correct? Especially HOME variable can be set to "/" instead of real home directory.
Pétur Ingi EgilssonSoftware Engineer -- Consultant

Commented:
Wget over ftp defaults to binary (i mode on ftp lingo), however, if you need to use ascii mode, you simply add ;type=a (without quotes) onto the end of the ftp url.

wget username:password@ftp://somedomain.com/remotefilename.tar.gz type=a
Pétur Ingi EgilssonSoftware Engineer -- Consultant

Commented:
There is no need to include the "close" in your .netrc

Author

Commented:
To run the job nightly I created a script file that sits in the /etc/cron.daily directory the netrc file is at /root/.netrc

Here is the crontab file

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

When I run from a command line I see that the .netrc file is sitting on
[root@Mark ~]# pwd
/root
[root@Mark ~]# ls -alh .netrc
-rwx------  1 root root 275 Jan  8 10:48 .netrc

[root@Mark cron.daily]# pwd
/etc/cron.daily
[root@Mark cron.daily]# ls -alh updateweb
-rwxr-xr-x  1 root root 3.8K Jan 18 17:17 updateweb

I am confused by the fact that the process works wonderfully interactively but only the FTP command in the script fails when ran by cron.




echo "############################################################################"
echo "## Webstore Nightly Maintainance                                          ##"
echo "##                                                                        ##"
echo "## Written By: Derek Hanson                                               ##"
echo "## Written On: 10/05/2004                                                 ##"
echo "##                                                                        ##"
echo "## Variable Descriptions                                                  ##"
echo "##                                                                        ##"
echo "##                                                                        ##"
echo "##                                                                        ##"
echo "## Log of Changes                                                         ##"
echo "## 10/05/2004 - Created Script                                            ##"
echo "## 01/18/2007 - removed export stop                                       ##"
echo "## 01/08/2008 - Added Functionality to update dev concurrently            ##"
echo "##                                                                        ##"
echo "############################################################################"
 
##################################################
## Variable Declarations                        ##
##################################################
echo "Setting Variable Declarations"
binpath=/usr/local/CI-live/CCI/bin
binpath_dev=/usr/local/CI-dev/CCI/bin
StoreFrontBin=/usr/local/StoreFrontBin
logpath=/usr/local/StoreFrontLogs
 
echo "############################################################################"
echo "############################################################################"
echo "##                                                                        ##"
echo "## The FTP client uses .netrc file to automate file copy from Levitcus.   ##"
echo "## If you want to bypass this feature use ftp -n remotesystem to open the ##"
echo "## connection.                                                            ##"
echo "##                                                                        ##"
echo "############################################################################"
echo "############################################################################"
 
echo "Beginning File Transfer"
/usr/bin/ftp levitcus
 
echo "Set Live Environment Variables and Stop Tomcat"
. $binpath/setenv.sh
. $binpath/tomcat stop
 
echo "Run Import Clear"
. $binpath/setenv.sh
. $binpath/importclear.sh
. $binpath_dev/setenv.sh
. $binpath_dev/importclear.sh
 
echo "Truncating Tables"
export PGPASSWORD="XXXXXXX"
$StoreFrontBin/pgsql/bin/psql ci-live -f $StoreFrontBin/truncatetables -U postgres
$StoreFrontBin/pgsql/bin/psql ci-dev -f $StoreFrontBin/truncatetables -U postgres
export PGPASSWORD=""
 
echo "Clear out Log Files"
rm -f $logpath/tc-start*
rm -f $logpath/dev/tc-start*
 
echo "Start Import"
. $binpath/setenv.sh
. $binpath/import.sh
. $binpath_dev/setenv.sh
. $binpath_dev/import.sh
 
echo "Start Tomcat Engine"
. $binpath/setenv.sh
. $binpath/tomcat start
 
echo "############################################################################"
echo "############################################################################"
echo "##                                                                        ##"
echo "## End of Nightly Update Web Job                                          ##"
echo "##                                                                        ##"
echo "############################################################################"
echo "############################################################################"

Open in new window

Top Expert 2005
Commented:
So it is the HOME variable problem.
Please reset it at the beginning of the /etc/cron.daily/updateweb script
export HOME="/root"
Pétur Ingi EgilssonSoftware Engineer -- Consultant

Commented:
Your .netrc must be in the same directory as the cronjob!
Your .netrc must have a BLANK line at the end.

Author

Commented:
I have added the export home to the update script and will update the question tomorrow.
Top Expert 2005

Commented:
> Your .netrc must be in the same directory as the cronjob!
untrue - HOME directory
> Your .netrc must have a BLANK line at the end.
possibly
Pétur Ingi EgilssonSoftware Engineer -- Consultant

Commented:
Looking at the man page for ~/.netrc i see:
-----
EXAMPLES

     Example 1: A Sample .netrc File

     A  .netrc file containing the following line:

     machine ray login demo password mypassword

     allows an autologin to the machine  ray using the login name
     demo with password mypassword.
------

Everything is in the same line..
Could it be the reason for your troubles?

Author

Commented:
Here are some more thoughts/facts

.netrc requires a blank line after every machine declaration as a seperator between machine declarations in the .netrc file.  

.netrc does not belong in the directory with the cron job, FTP uses the .netrc in the home directory of the user calling FTP. Since we are running a cron job the user is root and the file needs to be at /root/.netrc

This setup did work on a previous machine but when I have applied this methodology to other servers it has failed the FTP command.

I am trying the export of the Home variable to the the system variables but my question is this....

if FTP could not see the .netrc file then it would not know to process using the machine definition for levitcus.  If this is the case wouldn't I be getting a different error then "Login Failed"?

I will try to verify this interactively...
CERTIFIED EXPERT
Top Expert 2007

Commented:
Hi,

Why do you need to change the HOME env variable to / ?

Is the ..netrc file is in / or /root dir?

Can you change the content of .netrc to have only

machine levitcus
login XXXXXX
password XXXXXX

and change the script to:

#!/usr/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
/usr/bin/ftp levitcus << END
prompt off
lcd /usr/local/CI-live/CCI/postgresql/data/share/
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/postgresql/data/share/
cd astsfdtat
get SF400X.SF400X ImportData.txt
bye
END

Commented:
I'm having this very same issue and was able to resolve it with all the comments above. Thank you everyone for your various insights.

.netrc lives in the home directory and has the normal syntax as specified in the man page. The .netrc file isn't the issue. It's the fact that when the ftp script is run manually, HOME is defined whereas when the same script is run by cron, HOME isn't set. If HOME isn't set, the .netrc file can't be found and the login fails. Define the HOME environment variable and you should be good.

At the beginning of your script add HOME=<path>; export HOME. Make sure you export the variable.

Hopefully this will resolve your issue (Derek).

Jeff

Commented:
Further clarification:

1. Define the following in your .netrc:

machine levitcus
login XXXXXX
password XXXXXX

2. In some other file, define the rest of your script (script.ftp):

macdef init
prompt off
lcd /usr/local/CI-live/CCI/postgresql/data/share/
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/postgresql/data/share/
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit

3. Create another file which will be executed by cron (cron.sh):

HOME=<pathto.netrc>
export HOME
ftp -i levitcus < script.ftp

4. Now create the cron entry:

* * * * * sh <path>/cron.sh