?
Solved

How to stream data to an FTP server

Posted on 2009-12-18
11
Medium Priority
?
1,435 Views
Last Modified: 2013-12-22
Hello all,

I would like an advice.
I have a huge amount of data that must go to node C BUT through node A and B.
I have to be fast so I want the data not to be writen to nodes A and B.
Simply I am thinking of using A and B as the neccessary hoping nodes in order to reach node C.

Possible Solution 1:
Use NETTY HTTPFileServer change it in orer to immediately forwad all traffic to node A, B and from there use an FTP client (ftpj possibly or something similar) in order to write the data to node C. Node A and B would have a seperate Netty server installed in order to forward the data (without writing anything on the HDD). Something like tunneling.

Possible solution 2:
Make Java clients on node A and B (with Channels and streams) that would send all data to the destination node C (with the use of an ftp client again).

Possible solution 3:
Something off the self ? Maybe something native and not Java.
Is there any FTP sotware that basically "tunnels" or forwards all the data to the fial FTP server?
I have searched for FTP bridge but I have found irrelevant topics.

Is it possible to provide your solution to this problem ?
Is Netty really worths using instead of the Java new NIO (1.5 or 1.6)/The documentation is really not a lot. Have you found any good documentation except the user's guide that they have online?

As the FTP server I am thinking Filezilla server.
FreeNas or Filezlla do you think that could be faster ? (If I could ask such a question)

My aim is to transfer lot of files with Java to node C (ftp server)

Thank you in advance,

0
Comment
Question by:pouli
  • 6
  • 5
11 Comments
 
LVL 27

Expert Comment

by:Tolomir
ID: 26086150
I'm rather thinking about this idea:

http://en.wikipedia.org/wiki/Tunneling_protocol
0
 
LVL 27

Expert Comment

by:Tolomir
ID: 26086153
so this example could be adapted.

http://allthingsmarked.com/2006/08/17/how-to-set-up-a-windows-ssh-server-for-vnc-tunneling/

as far as I know you can link tunnels. But it requires you to install software on server a and b.


0
 

Author Comment

by:pouli
ID: 26086160
Hello Tolomir,

thank you for your answer.
Currently I have an application before node A (let's call it source) that produces the files. it writes them to the H.D.D.

I am thinking that now after the new change I should not write them to the H.D.D. but forwarding directly to the node A, etc...

Having these in mind ( I mean the fact that I can use Java API in order to stream data) how could I use the proposed solution ?

The wiki speak about the concept in general but I cannot see how it can fit in my probelm.

Could you please elaborate a little bit more about your idea ?
How you think that this will be possible ?

Thank you again.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 

Author Comment

by:pouli
ID: 26086165
Tolomir,

the posted link says:

"This tutorial will walk you through the steps to running a SSH server on your Windows machine and using it to create a secure tunnel through the Internet to use VNC"

Why vnc ?
This process is going to be automated by programs. No user interaction will be possible.
The source geernates data that automatically will be transfered to the target FTP server through node A and B.
0
 
LVL 27

Expert Comment

by:Tolomir
ID: 26086232
vnc is just an example. It could be any protocol.
0
 
LVL 27

Expert Comment

by:Tolomir
ID: 26086235
Ftp is port 21
0
 

Author Comment

by:pouli
ID: 26086264
This solution is going to be faster than the Java channels or use of the Netty project ?
By default SSH ise secured right ?
0
 
LVL 27

Accepted Solution

by:
Tolomir earned 150 total points
ID: 26088186
I found a tutorial to forward data from

source to a (ssh server) delivering to ftp server B. But I didn't find a 2 ( from source to A over B to C) hosts forwarder. So my suggestion might not even work.

this is the tutorial:

http://www.ssh.com/support/documentation/online/ssh/winhelp/32/tunneling.html#FIG-FORWARDINGTOATHIRDHOST

Additionally the connection from A to B is unencrypted. (Just the connection between source and server A is encrypted)

---

Advantage of the solution is, data is never stored on harddisk so this is of cause fast.

---

So my solution would only work (encrypted) if you could directly access server C from the source. If it were not ftp but ssh I could think of a connected ssh tunnel, but since ftp needs a control port and a second (data) port I was even unsure if ssh tunnels could deal with it.

this is a ssh chaining solution:

http://www.jedi.be/blog/2008/11/07/chaining-ssh-tunnels-easy-ssh-hopping/

0
 

Author Comment

by:pouli
ID: 26113668
Hello to Tolomir,

here is the solution that I was looking for.
I was looking for packages like the Apache's Java NIO framework.
I have seen many packages like this but Apache's was the "easiest to use".
I mean that I managed to change the included examples to something that worked for me.

I used an insecure channels for the hopping and the next step is to create a secure channel in order to finally copy the files to the destination.

The program easily saturated my locl LAN network speed (100Mbs) for 2500 files of 38 MB.
Basically I have transferred everything through one socket and at the end of each file I have appended a new byte array that I specify the location where the previous data should be written to.

Tolomir I wish you merry xmas and happy new year


0
 

Author Closing Comment

by:pouli
ID: 31668043
Reason has been specified inside the forum.
0
 
LVL 27

Expert Comment

by:Tolomir
ID: 26114526
Thank you very much. I wish you the same.
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

With the withdrawal of support for Windows Server 2003 this summer, many clients face the issue of moving away from their 2003 installs. There are a few options out there that many people/companies are selling. But the clients I have, haven't wanted…
In this post we will learn different types of Android Layout and some basics of an Android App.
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
Suggested Courses
Course of the Month17 days, 1 hour left to enroll

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question