How to make FTP on Ubuntu communicate via proxy server ?

How to make FTP client on Ubuntu communicate via proxy server ?


Client configuration:

declare -x ftp_proxy="http://10.224.3.95:8080/"
declare -x http_proxy="http://10.224.3.95:8080/"

http ,wget and apt-get  works .

However ftp refused communicate via proxy at all -  there is  nothing in tcpdump log !  

There is a log for wget :

CONNECT   Jun 11 19:19:51 [31506]: Connect (file descriptor 6): ip-10-224-3-78.ec2.internal [10.224.3.78]
CONNECT   Jun 11 19:19:51 [31506]: Request (file descriptor 6): GET http://www.google.com/ HTTP/1.1
INFO      Jun 11 19:19:51 [31506]: No upstream proxy for www.google.com
CONNECT   Jun 11 19:19:51 [31506]: Established connection to host "www.google.com" using file descriptor 7.
INFO      Jun 11 19:19:51 [31506]: Closed connection between local client (fd:6) and remote client (fd:7)

root@prod-sc-triad-master:/rna-data# wget www.google.com
--2015-06-11 19:18:56--  http://www.google.com/
Connecting to 10.224.3.95:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html.1'

    [ <=>                                                                                                                                                                                                                                ] 17,837      --.-K/s   in 0.01s

2015-06-11 19:18:57 (1.42 MB/s) - `index.html.1' saved [17837]
andrey_chevronAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

woolmilkporcCommented:
Some clients require the proxy variable being in uppercase.

Try

declare -x ftp_proxy="http://10.224.3.95:8080/"
declare -x FTP_PROXY=$ftp_proxy

or (same effect):
 
export ftp_proxy="http://10.224.3.95:8080/"
export FTP_PROXY=$ftp_proxy
0
andrey_chevronAuthor Commented:
Good idea - unfortunately no luck :
declare -x FTP_PROXY="http://10.224.3.95:8080/"

root@prod-sc-triad-master:/m-data# ftp -v -d mirrors.kernel.org
ftp: connect to address 198.145.20.143: Connection timed out
0
woolmilkporcCommented:
If I remember well the Ubuntu FTP client has the "set" subcommand.

Please start the client just with

ftp

then issue

set

What do you see next to "ftp_proxy" and "http_proxy" (the client should "see" your exported variables)?

If there's nothing you can still try setting the proxy temporarily to check if it's functional at all:

set ftp_proxy http://10.224.3.95:8080/
set http_proxy http://10.224.3.95:8080/
open mirrors.kernel.org

What do you see?

Maybe it could also be worth a try omitting the protocol part, i. e. "http://"
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

gheistCommented:
Your proxy refuses CONNECT command
Ask proxy admin to allow.
declare -x is same as export
0
andrey_chevronAuthor Commented:
Nope - ftp simply  ignored my settings:

On client side :

declare -x ftp_proxy="http://10.224.3.95:8080/"
declare -x http_proxy="http://10.224.3.95:8080/"

root@prod-sc-triad-master:/home/adousis# wget mirrors.kernel.org
--2015-06-15 12:05:30--  http://mirrors.kernel.org/
Connecting to 10.224.3.95:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 10316 (10K) [text/html]
Saving to: `index.html'

100%[===================================================================================================================================================================================================================================>] 10,316      --.-K/s   in 0s

2015-06-15 12:05:30 (176 MB/s) - `index.html' saved [10316/10316]


On server side :

root@prod-sc-triad-master:/home/adousis# ftp mirrors.kernel.org
^Croot@prod-sc-triad-master:/home/adousis# ^C
root@prod-sc-triad-master:/home/adousis# ^C
root@prod-sc-triad-master:/home/adousis# ftp
ftp> open
(to) mirrors.kernel.org
ftp: connect to address 198.145.20.143: Connection timed out
Trying 149.20.37.36...
ftp: connect: Connection timed out



root@starcluster-test-ubuntu:/var/log/tinyproxy# tail -f tinyproxy.log
INFO      Jun 15 06:38:12 [31509]: Adding Port [563] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31506]: Adding Port [563] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31507]: Disabling transmission of the "Via" header.
INFO      Jun 15 06:38:12 [31511]: Adding Port [563] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31512]: Adding Port [563] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31515]: Adding Port [563] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31513]: Adding Port [563] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31514]: Adding Port [563] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31507]: Adding Port [443] to the list allowed by CONNECT
INFO      Jun 15 06:38:12 [31507]: Adding Port [563] to the list allowed by CONNECT
CONNECT   Jun 15 12:06:37 [31508]: Connect (file descriptor 6): ip-10-224-3-78.ec2.internal [10.224.3.78]
CONNECT   Jun 15 12:06:37 [31508]: Request (file descriptor 6): GET http://mirrors.kernel.org/ HTTP/1.1
INFO      Jun 15 12:06:37 [31508]: No upstream proxy for mirrors.kernel.org
CONNECT   Jun 15 12:06:38 [31508]: Established connection to host "mirrors.kernel.org" using file descriptor 7.
INFO      Jun 15 12:06:38 [31508]: Closed connection between local client (fd:6) and remote client (fd:7)
0
andrey_chevronAuthor Commented:
Correction - "server side " should be in front of tail -f ... line
0
woolmilkporcCommented:
Did you try my suggestions in comment #40825227 above?
0
woolmilkporcCommented:
I just read "tinyproxy" in the posted "server side" data.

As far as I know the classic tinyproxy server does not support FTP!

At least you should have added port 21 in tinyproxy.conf by means of the "ConnectPort" directive, but even then I doubt that it will work.

There is a tinyproxy spin-off called tinyproxy-ex which offers FTP support:
https://github.com/tenchman/tinyproxy-ex
0
gheistCommented:
There is some tinyproxy extension that allows unencrypted-only passive-only FTP to work, like filezilla client etc.
0
andrey_chevronAuthor Commented:
Yes I tried -

root@prod-sc-triad-master:~# ftp
ftp> set
?Invalid command
0
andrey_chevronAuthor Commented:
I have installed tinyproxy-ex - however it did not help as a client still  not trying to communicate with proxy but keep trying to connect directly to FTP site  
0
woolmilkporcCommented:
So which is your FTP client?

Some clients do not use the proxy for interactive sessions but only for command-line fetches, like

ftp ftp://mirrors.kernel.org/path/to/file
(relative path)

or

ftp ftp://mirrors.kernel.org/%2fpath/to/file
(absolute path, if really needed)

Username/password/port can be supplied on the command line, if required
0
andrey_chevronAuthor Commented:
Client is Ubuntu 12.04 . We`re trying to use Perl module Net::FTP

BTW - wget is working via proxy :

root@prod-sc-triad-master:~# wget  ftp://mirrors.kernel.org/ubuntu/dists/wily/Release
--2015-06-15 15:56:03--  ftp://mirrors.kernel.org/ubuntu/dists/wily/Release
Connecting to 10.224.3.95:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 216879 (212K)
Saving to: `Release.1'

100%[===================================================================================================================================================>] 216,879      439K/s   in 0.5s

2015-06-15 15:56:05 (439 KB/s) - `Release.1' saved [216879/216879]

root@prod-sc-triad-master:~#
0
woolmilkporcCommented:
You never mentioned Net::FTP before. Is this a new approach, or did you just forget to tell us?
And why did you post FTP command line examples?

Net::FTP uses the environment variables "FTP_FIREWALL" and "FTP_FIREWALL_TYPE" instead of "FTP_PROXY".
And Net::FTP can only work with FTP proxies, not SOCKS or HTTP.
The URL you posted seems to refer to an HTTP proxy.  

Did you try the "command line only" (auto-fetch) method I suggested using the command line client?
0
gheistCommented:
ftp will connect using FTP
filezilla will use HTTP CONNECT
wget will use http:// request
0
andrey_chevronAuthor Commented:
We need both command line AND perl module - neither works

root@prod-sc-triad-master:/moderna-data# ftp ftp://mirrors.kernel.org:/ubuntu/dists/wily/Release
ftp: ftp://mirrors.kernel.org:/ubuntu/dists/wily/Release: Name or service not known

root@prod-sc-triad-master:/moderna-data# ftp ftp://mirrors.kernel.org/%2ubuntu/dists/wily/Release
ftp: ftp://mirrors.kernel.org/%2ubuntu/dists/wily/Release: Name or service not known
0
gheistCommented:
ftp command does not take URL (it does only on *BSD)
long ago you had ftp proxy where you logged in then logged in to remote site and then pumped files across.
0
andrey_chevronAuthor Commented:
filezilla and wget and curl - works .

I need to make ftp OR Net::Ftp work as well.  I have installed tinyproxy-ex - but see no difference....
0
woolmilkporcCommented:
As long as you're setting ftp_proxy to an HTTP URL you'll need an FTP command line client which is able to work with an HTTP proxy. Yours doesn't, obviously.
See the tinyproxy-ex docs how to enable FTP proxy and which address format to use for the "ftp_proxy" variable.

There is (BSD style) "tnftp" for Ubuntu which supports HTTP proxy and recognizes the URL format I suggested:
http://packages.ubuntu.com/en/trusty/tnftp

And anyway, Net::FTP doesn't support HTTP proxies at all.
0
gheistCommented:
ftp command cannot be fixed up to use HTTP CONNECT via proxy, nor to take ftp URLs
ftp:// urls already handled by wget/curl
if you need proxy CONNECT you need to rig filezilla with Xvfb for authentic FTP experience.
0
woolmilkporcCommented:
>> ftp command cannot be fixed up to use HTTP CONNECT via proxy,  <<

tnftp can. Quoting the man page:

ftp://[user[:password]@]host[:port]/path[/][;type=X]

An FTP URL, retrieved using the FTP protocol if set ftp_proxy isn't defined. Otherwise, transfer the URL using HTTP via the proxy defined in set ftp_proxy. If set ftp_proxy isn't defined and user is given, login as user. In this case, use password if supplied, otherwise prompt the user for one.
0
gheistCommented:
It is no different than wget.... it just sends ftp:// url to http:// server at ftp_proxy address...
0
woolmilkporcCommented:
>> It is no different than wget <<

Yes, but it is a full-featured FTP client nevertheless.
0
gheistCommented:
It does not use full features over proxy....
0
andrey_chevronAuthor Commented:
Yes - tnftp - works !

root@prod-sc-triad-master:/moderna-data# tnftp ftp://mirrors.kernel.org/ubuntu/dists/wily/Release
Requesting ftp://mirrors.kernel.org/ubuntu/dists/wily/Release
  (via 10.224.3.95:3128)
100% |************************************************************************************************************************************************|   211 KiB  521.08 KiB/s    00:00 ETA
216879 bytes retrieved in 00:00 (521.01 KiB/s)
0
andrey_chevronAuthor Commented:
If I may ask - is there a solution for Net:FTP ? Otherwise I have to change a perl script ....
0
woolmilkporcCommented:
Glad to hear that tnftp worked for you.

Net::FTP cannot work with an HTTP proxy.

There must be a proxy server which supports USER, PASS and possibly one of SITE or OPEN.

Find a list of supported methods (FTP_FIREWALL_TYPE) in the NET::Config docs:
http://search.cpan.org/~shay/libnet-3.06/lib/Net/Config.pm

You can try using your new tinyproxy-ex server, but I'm rather sure that it only offers FTP over HTTP
which is not supported by Net::FTP.

Anyway, before testing make sure to specify the correct port in the "Port" setting of Net::FTP.
This port is defined in tinyproxy-ex.conf by means of the "Listen" directive.

Here is the description of tinyproxy-ex.conf:
https://github.com/tenchman/tinyproxy-ex/blob/master/doc/tinyproxy-ex.conf.in

Here is the description of the Net::FTP Constructor (the place where PORT, FTP_FIREWALL and FTP_FIREWALL_TYPE live):
http://search.cpan.org/~shay/libnet-3.06/lib/Net/FTP.pm 

If Net::FTP fails to work over your proxy (which I expect, unfortunately) I'd suggest using LWP.
Here is a tutorial:
https://metacpan.org/pod/distribution/libwww-perl/lwptut.pod


Good luck!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
gheistCommented:
WWW::Curl does what you want - requests ftp:// URL via proxy
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux Networking

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.