.netrc works with FTP in an interactive mode but will not run during batch
I have a nightly job that runs on our webservers that uses FTP to download a build-file to our webserver.
In the cron job the command is issued as follows (where levitcus is the name of the server):
echo "Beginning File Transfer"
/usr/bin/ftp levitcus
The .netrc file contains the following script:
machine levitcus
login XXXXXX
password XXXXXX
macdef init
prompt off
lcd /usr/local/CI-live/CCI/pos
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/post
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit
When I pull the Logs from mail on the webserver I have this line when the FTP Command should be executed:
Login failed.
Thats it thats all I have no extra errors or anything. When I manually run the command: ftp levitcus everything runs beautifully.
I need help with this as I am really tired of manually triggering the ftp download from home in the wee hours of the morning.
Thanks
Derek
In the cron job the command is issued as follows (where levitcus is the name of the server):
echo "Beginning File Transfer"
/usr/bin/ftp levitcus
The .netrc file contains the following script:
machine levitcus
login XXXXXX
password XXXXXX
macdef init
prompt off
lcd /usr/local/CI-live/CCI/pos
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/post
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit
When I pull the Logs from mail on the webserver I have this line when the FTP Command should be executed:
Login failed.
Thats it thats all I have no extra errors or anything. When I manually run the command: ftp levitcus everything runs beautifully.
I need help with this as I am really tired of manually triggering the ftp download from home in the wee hours of the morning.
Thanks
Derek
ASKER
I have tried working with both wget and curl to download the file. The issue I have run into both times is that when the file is being downloaded from our AS400 (which builds this file and hosts the file for download) wget and curl were not able to navigate the FTP structure from the AS400.
I may be doing something wrong but the command I was using was this
wget ftp://USERNAME:PASSWORD@levitcus/astsfdta/SF400X.SF400X --passive-ftp --no-remove-listing -d
The file member name is SF400X in the object SF400X in the library ASTSFDTA. When wget processes the file the output is text file filled with @ and doesn't seem to ever find the end of the file (well until the HD was filled, I tried this and walked away, didn't work very well for me). curl gave me the exact same output problem.
I may be doing something wrong but the command I was using was this
wget ftp://USERNAME:PASSWORD@levitcus/astsfdta/SF400X.SF400X --passive-ftp --no-remove-listing -d
The file member name is SF400X in the object SF400X in the library ASTSFDTA. When wget processes the file the output is text file filled with @ and doesn't seem to ever find the end of the file (well until the HD was filled, I tried this and walked away, didn't work very well for me). curl gave me the exact same output problem.
How have You configured the nightly job? From cron? Sure the user cron entry is run as is correct? Especially HOME variable can be set to "/" instead of real home directory.
Wget over ftp defaults to binary (i mode on ftp lingo), however, if you need to use ascii mode, you simply add ;type=a (without quotes) onto the end of the ftp url.
wget username:password@ftp://somedomain.com/remotefilename.tar.gz type=a
wget username:password@ftp://somedomain.com/remotefilename.tar.gz type=a
There is no need to include the "close" in your .netrc
ASKER
To run the job nightly I created a script file that sits in the /etc/cron.daily directory the netrc file is at /root/.netrc
Here is the crontab file
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
When I run from a command line I see that the .netrc file is sitting on
[root@Mark ~]# pwd
/root
[root@Mark ~]# ls -alh .netrc
-rwx------ 1 root root 275 Jan 8 10:48 .netrc
[root@Mark cron.daily]# pwd
/etc/cron.daily
[root@Mark cron.daily]# ls -alh updateweb
-rwxr-xr-x 1 root root 3.8K Jan 18 17:17 updateweb
I am confused by the fact that the process works wonderfully interactively but only the FTP command in the script fails when ran by cron.
Here is the crontab file
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
When I run from a command line I see that the .netrc file is sitting on
[root@Mark ~]# pwd
/root
[root@Mark ~]# ls -alh .netrc
-rwx------ 1 root root 275 Jan 8 10:48 .netrc
[root@Mark cron.daily]# pwd
/etc/cron.daily
[root@Mark cron.daily]# ls -alh updateweb
-rwxr-xr-x 1 root root 3.8K Jan 18 17:17 updateweb
I am confused by the fact that the process works wonderfully interactively but only the FTP command in the script fails when ran by cron.
echo "############################################################################"
echo "## Webstore Nightly Maintainance ##"
echo "## ##"
echo "## Written By: Derek Hanson ##"
echo "## Written On: 10/05/2004 ##"
echo "## ##"
echo "## Variable Descriptions ##"
echo "## ##"
echo "## ##"
echo "## ##"
echo "## Log of Changes ##"
echo "## 10/05/2004 - Created Script ##"
echo "## 01/18/2007 - removed export stop ##"
echo "## 01/08/2008 - Added Functionality to update dev concurrently ##"
echo "## ##"
echo "############################################################################"
##################################################
## Variable Declarations ##
##################################################
echo "Setting Variable Declarations"
binpath=/usr/local/CI-live/CCI/bin
binpath_dev=/usr/local/CI-dev/CCI/bin
StoreFrontBin=/usr/local/StoreFrontBin
logpath=/usr/local/StoreFrontLogs
echo "############################################################################"
echo "############################################################################"
echo "## ##"
echo "## The FTP client uses .netrc file to automate file copy from Levitcus. ##"
echo "## If you want to bypass this feature use ftp -n remotesystem to open the ##"
echo "## connection. ##"
echo "## ##"
echo "############################################################################"
echo "############################################################################"
echo "Beginning File Transfer"
/usr/bin/ftp levitcus
echo "Set Live Environment Variables and Stop Tomcat"
. $binpath/setenv.sh
. $binpath/tomcat stop
echo "Run Import Clear"
. $binpath/setenv.sh
. $binpath/importclear.sh
. $binpath_dev/setenv.sh
. $binpath_dev/importclear.sh
echo "Truncating Tables"
export PGPASSWORD="XXXXXXX"
$StoreFrontBin/pgsql/bin/psql ci-live -f $StoreFrontBin/truncatetables -U postgres
$StoreFrontBin/pgsql/bin/psql ci-dev -f $StoreFrontBin/truncatetables -U postgres
export PGPASSWORD=""
echo "Clear out Log Files"
rm -f $logpath/tc-start*
rm -f $logpath/dev/tc-start*
echo "Start Import"
. $binpath/setenv.sh
. $binpath/import.sh
. $binpath_dev/setenv.sh
. $binpath_dev/import.sh
echo "Start Tomcat Engine"
. $binpath/setenv.sh
. $binpath/tomcat start
echo "############################################################################"
echo "############################################################################"
echo "## ##"
echo "## End of Nightly Update Web Job ##"
echo "## ##"
echo "############################################################################"
echo "############################################################################"
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Your .netrc must be in the same directory as the cronjob!
Your .netrc must have a BLANK line at the end.
Your .netrc must have a BLANK line at the end.
ASKER
I have added the export home to the update script and will update the question tomorrow.
> Your .netrc must be in the same directory as the cronjob!
untrue - HOME directory
> Your .netrc must have a BLANK line at the end.
possibly
untrue - HOME directory
> Your .netrc must have a BLANK line at the end.
possibly
Looking at the man page for ~/.netrc i see:
-----
EXAMPLES
Example 1: A Sample .netrc File
A .netrc file containing the following line:
machine ray login demo password mypassword
allows an autologin to the machine ray using the login name
demo with password mypassword.
------
Everything is in the same line..
Could it be the reason for your troubles?
-----
EXAMPLES
Example 1: A Sample .netrc File
A .netrc file containing the following line:
machine ray login demo password mypassword
allows an autologin to the machine ray using the login name
demo with password mypassword.
------
Everything is in the same line..
Could it be the reason for your troubles?
ASKER
Here are some more thoughts/facts
.netrc requires a blank line after every machine declaration as a seperator between machine declarations in the .netrc file.
.netrc does not belong in the directory with the cron job, FTP uses the .netrc in the home directory of the user calling FTP. Since we are running a cron job the user is root and the file needs to be at /root/.netrc
This setup did work on a previous machine but when I have applied this methodology to other servers it has failed the FTP command.
I am trying the export of the Home variable to the the system variables but my question is this....
if FTP could not see the .netrc file then it would not know to process using the machine definition for levitcus. If this is the case wouldn't I be getting a different error then "Login Failed"?
I will try to verify this interactively...
.netrc requires a blank line after every machine declaration as a seperator between machine declarations in the .netrc file.
.netrc does not belong in the directory with the cron job, FTP uses the .netrc in the home directory of the user calling FTP. Since we are running a cron job the user is root and the file needs to be at /root/.netrc
This setup did work on a previous machine but when I have applied this methodology to other servers it has failed the FTP command.
I am trying the export of the Home variable to the the system variables but my question is this....
if FTP could not see the .netrc file then it would not know to process using the machine definition for levitcus. If this is the case wouldn't I be getting a different error then "Login Failed"?
I will try to verify this interactively...
Hi,
Why do you need to change the HOME env variable to / ?
Is the ..netrc file is in / or /root dir?
Can you change the content of .netrc to have only
machine levitcus
login XXXXXX
password XXXXXX
and change the script to:
#!/usr/bin/bash
PATH=/sbin:/bin:/usr/sbin:
MAILTO=root
/usr/bin/ftp levitcus << END
prompt off
lcd /usr/local/CI-live/CCI/pos
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/post
cd astsfdtat
get SF400X.SF400X ImportData.txt
bye
END
Why do you need to change the HOME env variable to / ?
Is the ..netrc file is in / or /root dir?
Can you change the content of .netrc to have only
machine levitcus
login XXXXXX
password XXXXXX
and change the script to:
#!/usr/bin/bash
PATH=/sbin:/bin:/usr/sbin:
MAILTO=root
/usr/bin/ftp levitcus << END
prompt off
lcd /usr/local/CI-live/CCI/pos
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/post
cd astsfdtat
get SF400X.SF400X ImportData.txt
bye
END
I'm having this very same issue and was able to resolve it with all the comments above. Thank you everyone for your various insights.
.netrc lives in the home directory and has the normal syntax as specified in the man page. The .netrc file isn't the issue. It's the fact that when the ftp script is run manually, HOME is defined whereas when the same script is run by cron, HOME isn't set. If HOME isn't set, the .netrc file can't be found and the login fails. Define the HOME environment variable and you should be good.
At the beginning of your script add HOME=<path>; export HOME. Make sure you export the variable.
Hopefully this will resolve your issue (Derek).
Jeff
.netrc lives in the home directory and has the normal syntax as specified in the man page. The .netrc file isn't the issue. It's the fact that when the ftp script is run manually, HOME is defined whereas when the same script is run by cron, HOME isn't set. If HOME isn't set, the .netrc file can't be found and the login fails. Define the HOME environment variable and you should be good.
At the beginning of your script add HOME=<path>; export HOME. Make sure you export the variable.
Hopefully this will resolve your issue (Derek).
Jeff
Further clarification:
1. Define the following in your .netrc:
machine levitcus
login XXXXXX
password XXXXXX
2. In some other file, define the rest of your script (script.ftp):
macdef init
prompt off
lcd /usr/local/CI-live/CCI/pos
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/post
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit
3. Create another file which will be executed by cron (cron.sh):
HOME=<pathto.netrc>
export HOME
ftp -i levitcus < script.ftp
4. Now create the cron entry:
* * * * * sh <path>/cron.sh
1. Define the following in your .netrc:
machine levitcus
login XXXXXX
password XXXXXX
2. In some other file, define the rest of your script (script.ftp):
macdef init
prompt off
lcd /usr/local/CI-live/CCI/pos
cd astsfdta
get SF400X.SF400X ImportData.txt
lcd /usr/local/CI-dev/CCI/post
cd astsfdtat
get SF400X.SF400X ImportData.txt
close
quit
3. Create another file which will be executed by cron (cron.sh):
HOME=<pathto.netrc>
export HOME
ftp -i levitcus < script.ftp
4. Now create the cron entry:
* * * * * sh <path>/cron.sh
wget --ftp-user=XXXX --ftp-password=XXXX ftp://SERVERNAME/FILENAME