.netrc works with FTP in an interactive mode but will not run during batch

I have a nightly job that runs on our webservers that uses FTP to download a build-file to our webserver.  

In the cron job the command is issued as follows (where levitcus is the name of the server):
echo "Beginning File Transfer"
/usr/bin/ftp levitcus

The .netrc file contains the following script:
machine levitcus
login XXXXXX
password XXXXXX
macdef init
prompt off
lcd /usr/local/CI-live/CCI/postgresql/data/share/
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/postgresql/data/share/
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit

When I pull the Logs from mail on the webserver I have this line when the FTP Command should be executed:
Login failed.

Thats it thats all I have no extra errors or anything.  When I manually run the command: ftp levitcus everything runs beautifully.

I need help with this as I am really tired of manually triggering the ftp download from home in the wee hours of the morning.

Thanks
Derek
RSchnabelAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Pétur Ingi EgilssonSoftware Engineer -- ConsultantCommented:
You can use wget instead of ftp to fetch from FTP's.

wget --ftp-user=XXXX --ftp-password=XXXX ftp://SERVERNAME/FILENAME
0
RSchnabelAuthor Commented:
I have tried working with both wget and curl to download the file.  The issue I have run into both times is that when the file is being downloaded from our AS400 (which builds this file and hosts the file for download) wget and curl were not able to navigate the FTP structure from the AS400.  

I may be doing something wrong but the command I was using was this

wget ftp://USERNAME:PASSWORD@levitcus/astsfdta/SF400X.SF400X --passive-ftp --no-remove-listing -d

The file member name is SF400X in the object SF400X in the library ASTSFDTA.  When wget processes the file the output is text file filled with @ and doesn't seem to ever find the end of the file (well until the HD was filled, I tried this and walked away, didn't work very well for me).  curl gave me the exact same output problem.
0
ravenplCommented:
How have You configured the nightly job? From cron? Sure the user cron entry is run as is correct? Especially HOME variable can be set to "/" instead of real home directory.
0
Newly released Acronis True Image 2019

In announcing the release of the 15th Anniversary Edition of Acronis True Image 2019, the company revealed that its artificial intelligence-based anti-ransomware technology – stopped more than 200,000 ransomware attacks on 150,000 customers last year.

Pétur Ingi EgilssonSoftware Engineer -- ConsultantCommented:
Wget over ftp defaults to binary (i mode on ftp lingo), however, if you need to use ascii mode, you simply add ;type=a (without quotes) onto the end of the ftp url.

wget username:password@ftp://somedomain.com/remotefilename.tar.gz type=a
0
Pétur Ingi EgilssonSoftware Engineer -- ConsultantCommented:
There is no need to include the "close" in your .netrc
0
RSchnabelAuthor Commented:
To run the job nightly I created a script file that sits in the /etc/cron.daily directory the netrc file is at /root/.netrc

Here is the crontab file

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

When I run from a command line I see that the .netrc file is sitting on
[root@Mark ~]# pwd
/root
[root@Mark ~]# ls -alh .netrc
-rwx------  1 root root 275 Jan  8 10:48 .netrc

[root@Mark cron.daily]# pwd
/etc/cron.daily
[root@Mark cron.daily]# ls -alh updateweb
-rwxr-xr-x  1 root root 3.8K Jan 18 17:17 updateweb

I am confused by the fact that the process works wonderfully interactively but only the FTP command in the script fails when ran by cron.




echo "############################################################################"
echo "## Webstore Nightly Maintainance                                          ##"
echo "##                                                                        ##"
echo "## Written By: Derek Hanson                                               ##"
echo "## Written On: 10/05/2004                                                 ##"
echo "##                                                                        ##"
echo "## Variable Descriptions                                                  ##"
echo "##                                                                        ##"
echo "##                                                                        ##"
echo "##                                                                        ##"
echo "## Log of Changes                                                         ##"
echo "## 10/05/2004 - Created Script                                            ##"
echo "## 01/18/2007 - removed export stop                                       ##"
echo "## 01/08/2008 - Added Functionality to update dev concurrently            ##"
echo "##                                                                        ##"
echo "############################################################################"
 
##################################################
## Variable Declarations                        ##
##################################################
echo "Setting Variable Declarations"
binpath=/usr/local/CI-live/CCI/bin
binpath_dev=/usr/local/CI-dev/CCI/bin
StoreFrontBin=/usr/local/StoreFrontBin
logpath=/usr/local/StoreFrontLogs
 
echo "############################################################################"
echo "############################################################################"
echo "##                                                                        ##"
echo "## The FTP client uses .netrc file to automate file copy from Levitcus.   ##"
echo "## If you want to bypass this feature use ftp -n remotesystem to open the ##"
echo "## connection.                                                            ##"
echo "##                                                                        ##"
echo "############################################################################"
echo "############################################################################"
 
echo "Beginning File Transfer"
/usr/bin/ftp levitcus
 
echo "Set Live Environment Variables and Stop Tomcat"
. $binpath/setenv.sh
. $binpath/tomcat stop
 
echo "Run Import Clear"
. $binpath/setenv.sh
. $binpath/importclear.sh
. $binpath_dev/setenv.sh
. $binpath_dev/importclear.sh
 
echo "Truncating Tables"
export PGPASSWORD="XXXXXXX"
$StoreFrontBin/pgsql/bin/psql ci-live -f $StoreFrontBin/truncatetables -U postgres
$StoreFrontBin/pgsql/bin/psql ci-dev -f $StoreFrontBin/truncatetables -U postgres
export PGPASSWORD=""
 
echo "Clear out Log Files"
rm -f $logpath/tc-start*
rm -f $logpath/dev/tc-start*
 
echo "Start Import"
. $binpath/setenv.sh
. $binpath/import.sh
. $binpath_dev/setenv.sh
. $binpath_dev/import.sh
 
echo "Start Tomcat Engine"
. $binpath/setenv.sh
. $binpath/tomcat start
 
echo "############################################################################"
echo "############################################################################"
echo "##                                                                        ##"
echo "## End of Nightly Update Web Job                                          ##"
echo "##                                                                        ##"
echo "############################################################################"
echo "############################################################################"

Open in new window

0
ravenplCommented:
So it is the HOME variable problem.
Please reset it at the beginning of the /etc/cron.daily/updateweb script
export HOME="/root"
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Pétur Ingi EgilssonSoftware Engineer -- ConsultantCommented:
Your .netrc must be in the same directory as the cronjob!
Your .netrc must have a BLANK line at the end.
0
RSchnabelAuthor Commented:
I have added the export home to the update script and will update the question tomorrow.
0
ravenplCommented:
> Your .netrc must be in the same directory as the cronjob!
untrue - HOME directory
> Your .netrc must have a BLANK line at the end.
possibly
0
Pétur Ingi EgilssonSoftware Engineer -- ConsultantCommented:
Looking at the man page for ~/.netrc i see:
-----
EXAMPLES

     Example 1: A Sample .netrc File

     A  .netrc file containing the following line:

     machine ray login demo password mypassword

     allows an autologin to the machine  ray using the login name
     demo with password mypassword.
------

Everything is in the same line..
Could it be the reason for your troubles?
0
RSchnabelAuthor Commented:
Here are some more thoughts/facts

.netrc requires a blank line after every machine declaration as a seperator between machine declarations in the .netrc file.  

.netrc does not belong in the directory with the cron job, FTP uses the .netrc in the home directory of the user calling FTP. Since we are running a cron job the user is root and the file needs to be at /root/.netrc

This setup did work on a previous machine but when I have applied this methodology to other servers it has failed the FTP command.

I am trying the export of the Home variable to the the system variables but my question is this....

if FTP could not see the .netrc file then it would not know to process using the machine definition for levitcus.  If this is the case wouldn't I be getting a different error then "Login Failed"?

I will try to verify this interactively...
0
omarfaridCommented:
Hi,

Why do you need to change the HOME env variable to / ?

Is the ..netrc file is in / or /root dir?

Can you change the content of .netrc to have only

machine levitcus
login XXXXXX
password XXXXXX

and change the script to:

#!/usr/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
/usr/bin/ftp levitcus << END
prompt off
lcd /usr/local/CI-live/CCI/postgresql/data/share/
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/postgresql/data/share/
cd astsfdtat
get SF400X.SF400X ImportData.txt
bye
END
0
tdiopsCommented:
I'm having this very same issue and was able to resolve it with all the comments above. Thank you everyone for your various insights.

.netrc lives in the home directory and has the normal syntax as specified in the man page. The .netrc file isn't the issue. It's the fact that when the ftp script is run manually, HOME is defined whereas when the same script is run by cron, HOME isn't set. If HOME isn't set, the .netrc file can't be found and the login fails. Define the HOME environment variable and you should be good.

At the beginning of your script add HOME=<path>; export HOME. Make sure you export the variable.

Hopefully this will resolve your issue (Derek).

Jeff
0
tdiopsCommented:
Further clarification:

1. Define the following in your .netrc:

machine levitcus
login XXXXXX
password XXXXXX

2. In some other file, define the rest of your script (script.ftp):

macdef init
prompt off
lcd /usr/local/CI-live/CCI/postgresql/data/share/
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/postgresql/data/share/
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit

3. Create another file which will be executed by cron (cron.sh):

HOME=<pathto.netrc>
export HOME
ftp -i levitcus < script.ftp

4. Now create the cron entry:

* * * * * sh <path>/cron.sh
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.