Solved

lftp (sftp through proxy) won't connect when using cron

Posted on 2011-09-13
14
2,603 Views
Last Modified: 2012-05-12
Hi Experts

I have a testing problem. I am mirroring some files with lftp from an akamai host to a local server. I am connecting using sftp via a SOCKS proxy and using cron to automate the job.

My problem is when the script is run by cron I cannot connect to host. I can run the script manually and connect to akamai by command line (using user root and the crontab is installed for root) but I am not sure, when using cron, it is picking up the socks proxyconnect command from ssh_config.

Has anyone come across this problem before?

Many thanks
0
Comment
Question by:Taniap
  • 6
  • 5
  • 3
14 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36530065
Check the environment!

cron doesn't run shell initialization profiles, so you might be missing some required environment variable, or a PATH.

wmp
0
 

Author Comment

by:Taniap
ID: 36530412
hmmm, I have
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>

everything has a defined path in the cron and the script
I also have another lftp script automated through cron - again mirroring a server which has been running for ages with no problems - the only difference is it is not using a proxy to connect.

I am a bit stumped
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36530543
Do I understand you right that cron uses the wrong "ProxyCommand"?

Well, you can pass ssh options to sftp, like

sftp -o ProxyCommand='my proxy command string' ... ... ...

 

0
 

Author Comment

by:Taniap
ID: 36530685
I'm not sure but that's what I'm suspecting.

my ~/.ssh/ssh_config has the following
Host xxx.xxx.akamai.com
   ProxyCommand connect -S socks.xxx.xxx.xx.xx:1080 %h %p

my cron is:
5 **** /usr/bin/lftp -f /root/download.x

and download.x is:
debug -o /root/debug.txt 1
open -u xxx,xxx -p 22 sftp://xxx.xxx.akamai.com
...plus all the mirroring commands  

The debug.txt is just saying ****peer closed connection
0
 
LVL 9

Expert Comment

by:parparov
ID: 36530834
You should add the same entry in ~root/.ssh/ssh_config for Host xxx.xxx.akamai.com
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36530929
I hope that you have spaces between the asterisks in crontab ( 5 * * * * )!

You could indeed try to add the ProxCommand to the crontab entry, but please be aware that under cron you must escape the percent signs ( % ) with a backslash ( \% ), because they have a special meaning there (start of stdin).

Like this:

5 * * * * /usr/bin/lftp -o ProxyCommand='connect -S socks.xxx.xxx.xx.xx:1080 \%h \%p' -f /root/download.x


0
 

Author Comment

by:Taniap
ID: 36531251
Thanks I'll give that a go tomorrow - I've given up for tonight

(and yep, there are spaces between the asterisks - I didn't cut & paste just typed it quickly...)
Many thanks for you input with this
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36556853
Did you get any further with this issue?
0
 

Author Comment

by:Taniap
ID: 36559299
Hi Woolmilkporc

No I'm still having issues with it. I tried:

5 * * * * /usr/bin/lftp -o ProxyCommand='connect -S socks.xxx.xxx.xx.xx:1080 \%h \%p' -f /root/download.x

lftp: invalid option -- 'o'


5 * * * * /usr/bin/lftp -c ProxyCommand='connect -S socks.xxx.xxx.xx.xx:1080 \%h \%p' -f /root/download.x

connection refused

5 * * * * /usr/bin/lftp set ftp:proxy='socks.xxx.xxx.xx.xx:1080 \%h \%p' -f /root/download.x

lftp: invalid option -- 'f'

I then reverted back to:
5 **** /usr/bin/lftp -f /root/download.x

and amended download.x to:
debug -o /root/debug.txt 5
set ftp:proxy=socks.xxx.xxx.xxx.xxx:1080
set ftp:proxy-auth-type=open
open -u xxx,xxx -p 22 sftp://xxx.xxx.akamai.com
...plus all the mirroring commands

running lftp -f /root/download.x as root from command line works but from cron once again - connection refused in the debug.txt  

..now there must be a way to get to cron to use the proxy but I can't for the life in me find it!!!

0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 36559514

It find it strange that it should work from command line because I suspect that the option "-u xxx,xxx" does not work with lftp as an sftp client.

What is "xxx,xxx"? root and his password?

If there is public key authentication configured you could try

open -p 22 sftp://xxx@xxx.xxx.akamai.com

with xxx being the username from "-u xxx,..."

Again, please remember that passwordless login via public key must be possible.

wmp


0
 
LVL 9

Expert Comment

by:parparov
ID: 36560499
Did you alter the root .ssh/config file as proposed?
0
 

Author Comment

by:Taniap
ID: 36560532
hi parparov

yes

my root/.ssh/ssh_config has the following
Host xxx.xxx.akamai.com
   ProxyCommand connect -S socks.xxx.xxx.xx.xx:1080 %h %p

and has been from the start - hence why it's working from the command line OK

Just having trouble getting it to work when using root's cron
0
 
LVL 9

Accepted Solution

by:
parparov earned 500 total points
ID: 36560567
whoops, the file should be ~root/.ssh/config and not ~root/.ssh/ssh_config
0
 

Author Comment

by:Taniap
ID: 36598298
Hi parparov

I created a root/.ssh/config and it now looks like it's working sort of ... (I'm getting authentication errors - OK if run manually but failing via cron - but that is another investigation - at least it's getting through the proxy now).

Many thanks for your help
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now