internet file transfer

hi,
I was just wondering how yahoo makes it possible for com1 and com2 to connect to each other? The reason I am asking this is because I have created small network application, which main purpose is to transfer a file from com1 to com2, and unless I open/forward the port on com2, com1 is unable to connect to it. Both PCs use same mini DLS routers.

Yahoo has no problem at all when I do the same task because it creates direct connection but, I have no idea how... does it use some sort of network address translation internally or there is something else to it..?
LVL 13
davidlars99Asked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
Ravi SinghConnect With a Mentor Senior Software EngineerCommented:
Hi david,

The server would typically store the current IP's of the "online" peers, so once a peer has authenticated on the server, the server would probably send the authenticated peer a list of the online peers with their proper "global" ip addresses, thus allowing peers to establish direct connections with other peers.

Peer2 can accept the packet because the NAT gateway maintains a list of return addresses which it assigns to packets on their way out of the local network (i.e. if peer2 sends a packet to the server - the packet is first intercepted by the router - at this point the packets return address is the local peer computers' ipaddress + the port its listening on - the router changes the return address to the ip address of the router + an available port number on the router - it stores this info in a local lookup table - so when the packet eventually arrives at the server its return address is the routers ip:port number - the server would then send this to peer1 (for example when peer1 authenticates on the peer network) - peer1 can then send a packet directly to that ipaddress+port number of the router (which peer2 is under), the router performs the lookup and notices that the incoming packet has the destination address of the one assigned by the router as a return address - the router then forwards this packet to peer2 on the port number which the originating software was listening on.

So the packet goes to the port which the software was originally listening on, I guess its then upto you to perform some checks to see whether it is actually a packet that your listening for (using some secret headers etc in the data).
0
 
Ravi SinghSenior Software EngineerCommented:
Hi, not 100% sure if yahoo messenger supports this, but a lot of the peer-to-peer software today uses the UPnP protocol. Typically routers support this as a feature and allow applications to programmatically configure ports...

More Info - http://en.wikipedia.org/wiki/Upnp
C# Implementation - http://www.pixvillage.com/blogs/devblog/archive/2005/03/31/183.aspx
0
 
davidlars99Author Commented:
hi Zephyr__, I assumed so but, I don't think yahoo uses UPnP because it is disabled on my router. Can you still benefit from UPnP if it's disabled?
0
Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
Ravi SinghSenior Software EngineerCommented:
Hi again, doubt its usable if its disabled, could it be the case that them ports are already open (predefined port mappings)? Yahoo Messenger uses the following ports in case you want to check:

5000 (TCP - Chat)
5001 (TCP - Chat)
5055 (UDP - Phone)
5100 (TCP - Webcam*)

from: http://support.dlink.com/faq/view.asp?prod_id=1428
0
 
davidlars99Author Commented:
yes I know it does but, I have a DSL mini-router and as far as I know there has to be port forwarding enabled for those port that yahoo uses...  

I'll check if my router has pre-configured forwarding for yahoo ports, and let you know.
0
 
davidlars99Author Commented:
by the way, I have Westell 6600 router
0
 
Ravi SinghSenior Software EngineerCommented:
I can't find any specific information relating to how yahoo messenger overcomes problems related to NAT (in terms of direct p2p connections), but I have in the past come across various ways to bypass NAT to create p2p friendly applications (some of them involve the use of an intermediate server to "introduce peers". Some of the methods used are explained on the webpages below, they should give you a better idea:

NAT and Peer-to-peer networking:
http://alumnus.caltech.edu/~dank/peer-nat.html

How to communicate peer-to-peer through NAT (Network Address Translation):
http://www.mindcontrol.org/~hplus/nat-punch.html

Building NAT-Friendly Applications:
http://www.intel.com/cd/ids/developer/asmo-na/eng/79524.htm

Hope this helps.
0
 
davidlars99Author Commented:
Zephyr__, will this examples work if there's a intermediate server, which introduces peers to each other but both computers need to have port forwarding done?

I am confused...
0
 
Ravi SinghSenior Software EngineerCommented:
Hi,

Using the server to introduce the peers eliminates the need for port forwarding on the gateway (router) devices. So if for example peer1 wanted to communicate with peer2 (and both are behind a NAT enabled gateway) then using the approach explained in the second link (http://www.mindcontrol.org/~hplus/nat-punch.html)... peer1 would send a packet to the server, in order for the packet to reach the server it would go through the local NAT enabled gateway, the gateway would inject a "return address" into the packet (based on some unused port) and send the packet onto the server. The server would get the packet, note the return address for the packet and send the return address to peer2, peer2 can then communicate to peer1 using the return address, when peer2 sends a packet to peer1 on the return address... the NAT device on peer1 would recognise the return address should map to peer1 on the local network and forwards the packet to peer1...

The above was a simplified explanation, if you read the the webpage mentioned, it explains it in more detailed (pay extra attention to the section "Introducing the Introducer")... yahoo may or may not support this approach, but its one known way of bypassing NAT's limitations for p2p communication

Hope this helps
0
 
davidlars99Author Commented:
I read that section about 10 times...  :-)

what I don't understand are two things:
  1.  how can peer1 send a packet to peer2 without properly connecting to it, using Socket.Connect(EndPoint)?
  2.  how can peer2 accept a packet from a stranger such as peer1, when peer2 has never seen peer1's IP address?

or maybe I should cut the crap and just try it out?
0
 
davidlars99Author Commented:
ok finally got it to work. tested on DLS connection, where two computers have same exact speed (768 kb/sec) so, 36.1 MB file took exactly 29 minutes to transfer from peer1 to peer2, which is exactly 10 minutes faster than yahoo file transfer on both same computers...  ;)

[SERVER SOCKET]

using System;
using System.IO;
using System.Net;
using System.Text;
using System.Net.Sockets;
using System.Threading;

namespace ServerSocket
{
      class EntryPoint
      {
            static Socket cs;
            static Socket ss;            
            const int MAX_CONN = 1;
            const int MAX_BUFFER_SIZE = 66560; // tested on DSL connection
            const int MIN_BUFFER_SIZE = 1024;

            static FileStream buffer;

            static int totalBytes = 0;
            static int bytesRcvd = 0;
            static byte[] temp_buffer;

            static AsyncCallback receiveCallback;
            static AsyncCallback acceptCallback;

            [STAThread]
            static void Main(string[] args)
            {
                  receiveCallback = new AsyncCallback(ReceiveCallback);
                  acceptCallback = new AsyncCallback(AcceptCallback);

                  try
                  {
                        IPEndPoint ipe = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 3535);
                        ss = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                        ss.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);
                        ss.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveBuffer, MAX_BUFFER_SIZE);
                        ss.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.Debug, 0);
                        ss.Bind(ipe);
                        ss.Listen(MAX_CONN);
                        
                        while (true)
                        {
                              Console.WriteLine("Listening on - "+ipe.ToString()+"\n");
                              IAsyncResult acceptDone = ss.BeginAccept(acceptCallback, null);
                              acceptDone.AsyncWaitHandle.WaitOne();                              

                              string x = Console.ReadLine().Trim().ToLower();
                              if (x=="")
                                    break;
                        }
                  }
                  catch (Exception ex)
                  {
                        Console.WriteLine(ex.Message);
                  }
                  finally
                  {
                        if (ss.Connected)
                              ss.Close();            
                        
                        GC.Collect();
                  }                  
                  Console.WriteLine("All Sockets have been disposed!");
                  Console.ReadLine();
            }
            static void AcceptCallback(IAsyncResult ar)
            {
                  try
                  {
                        cs = ss.EndAccept(ar);            
                        ar.AsyncWaitHandle.Close();
                        cs.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveBuffer, MAX_BUFFER_SIZE);
                        cs.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.SendBuffer, MIN_BUFFER_SIZE);
                        cs.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.Debug, 0);
                        Console.WriteLine("Remote client - "+cs.RemoteEndPoint+"\n");            

                        buffer = new FileStream("new.txt", FileMode.Create, FileAccess.Write, FileShare.None);       
                        temp_buffer = new byte[MAX_BUFFER_SIZE];                  
                        cs.BeginReceive(temp_buffer, 0, MAX_BUFFER_SIZE, SocketFlags.None, receiveCallback, null);
                  }
                  catch (Exception ex)
                  {
                        Console.WriteLine(ex.Message);
                  }
            }
            static void ReceiveCallback(IAsyncResult ar)
            {
                  try
                  {
                        bytesRcvd = cs.EndReceive(ar);      
                        ar.AsyncWaitHandle.Close();
                        if (bytesRcvd>0)
                        {
                              unchecked { totalBytes += bytesRcvd; }
                              buffer.Write(temp_buffer, 0, bytesRcvd);                                    
                              temp_buffer = new byte[MAX_BUFFER_SIZE];
                              cs.BeginReceive(temp_buffer, 0, MAX_BUFFER_SIZE, SocketFlags.None, receiveCallback, null);
                        }
                        else
                        {                                                
                              if (cs.Connected)
                              {
                                    cs.Shutdown(SocketShutdown.Receive);
                                    cs.Close();
                              }
                              Console.WriteLine("Total data received:  "+Size(totalBytes));                                          
                              buffer.Close();      
                        }
                  }
                  catch (Exception ex)
                  {
                        Console.WriteLine(ex.Message);
                        if (cs.Connected)
                              cs.Close();                  
                  }
            }
            static string Size(int number)
            {
                  int mb = 1048576;            
                  if(number>mb)            
                        return String.Format("{0:0.0 mb}", Convert.ToDecimal(number)/mb);                  
                  else                  
                        return String.Format("{0:0.0 kb}", Convert.ToDecimal(number)/1024);                  
            }
      }
}




[CLIENT SOCKET]

using System;
using System.IO;
using System.Net;
using System.Text;
using System.Net.Sockets;
using System.Threading;
using System.Windows.Forms;

namespace ClientSocket
{
      class EntryPoint
      {
            static Socket s;
            const int MAX_BUFFER_SIZE = 66560; // tested on DSL connection
            const int MIN_BUFFER_SIZE = 1024;
            static int bytesLeft = MAX_BUFFER_SIZE;

            static FileStream fs = null;
            static byte[] fs_buffer;
            static int fs_bytesRead = 0;
            static int fs_length = 0;
            static int dataSent = 0;
            
            static ManualResetEvent connDone;
            static ManualResetEvent threadRun;

            [STAThread]
            static void Main(string[] args)
            {
                  connDone = new ManualResetEvent(false);
                  threadRun = new ManualResetEvent(false);
                  OpenFileDialog dialog = null;
                  try
                  {
                        dialog = new OpenFileDialog();                                                
                        if (dialog.ShowDialog()==DialogResult.OK)
                        {
                              IPEndPoint ipe = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 3535);
                              s = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                              s.Blocking = true;
                              s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.SendBuffer, MAX_BUFFER_SIZE);
                              s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveBuffer, MIN_BUFFER_SIZE);      
                              s.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.Debug, 0);
                              s.BeginConnect(ipe, new AsyncCallback(ConnectCallback), ipe);
                              connDone.WaitOne();

                              fs = File.OpenRead(dialog.FileName);                                                                                    
                              Console.WriteLine("Data Transmission in process - "+DateTime.Now+"\n");
                              fs_length = (int)fs.Length;      

                              Thread t = new Thread(new ThreadStart(Read_Send));
                              t.Start();                              
                              threadRun.WaitOne();                                                            
                        }
                  }      
                  catch (Exception ex)
                  {
                        Console.WriteLine(ex.Message);
                  }
                  finally
                  {
                        if (s.Connected) s.Close();      
                        if (dialog!=null) dialog.Dispose();
                        if (fs!=null) fs.Close();

                        GC.Collect();
                  }
                  Console.WriteLine("Data Transmission completed - "+DateTime.Now+"\n");
                  Console.WriteLine("Total data sent:  "+Size(dataSent));
                  Console.ReadLine();
            }
            static void ConnectCallback(IAsyncResult ar)
            {
                  try
                  {
                        IPEndPoint ipe = (IPEndPoint)ar.AsyncState;
                        s.EndConnect(ar);
                        ar.AsyncWaitHandle.Close();      
                        Console.WriteLine("Remote host - "+ipe.ToString()+"\n");
                  }
                  catch (Exception ex)
                  {
                        Console.WriteLine(ex.Message);
                  }
                  finally
                  {
                        connDone.Set();      
                  }
            }
            static void Read_Send()
            {
                  unchecked
                  {
                        try
                        {
                              while (dataSent<fs_length)
                              {                        
                                    if ((fs_bytesRead+MAX_BUFFER_SIZE)>fs_length)
                                          bytesLeft = fs_length-fs_bytesRead;      
                        
                                    fs_buffer = new byte[bytesLeft];
                                    fs.Read(fs_buffer, 0, bytesLeft);      
                                    fs_bytesRead += bytesLeft;

                                    dataSent += s.Send(fs_buffer, 0, bytesLeft, SocketFlags.None);
                              }
                        }
                        catch (Exception ex)
                        {
                              Console.WriteLine(ex.Message);
                        }
                        finally
                        {
                              Thread.Sleep(TimeSpan.FromSeconds(1));
                              if (s.Connected)
                                    s.Shutdown(SocketShutdown.Send);
                              threadRun.Set();      
                        }
                  }
            }
            static string Size(int number)
            {
                  int mb = 1048576;            
                  if(number>mb)            
                        return String.Format("{0:0.0 mb}", Convert.ToDecimal(number)/mb);                  
                  else                  
                        return String.Format("{0:0.0 kb}", Convert.ToDecimal(number)/1024);                  
            }
      }
}

0
 
davidlars99Author Commented:
Server socket is asyncronious and client socket is syncronious blocking. I couldn't get asyncronious client socket to work with files over 100 MB. I was getting System.StackOverflowException.
0
 
Ravi SinghSenior Software EngineerCommented:
Not 100% sure on what the cause could be, but if the exception is occuring in relation to how large the file being transferred is then i'd guess that the callbacks are causing the problem... have you tried increasing the buffer sizes?



0
 
Ravi SinghSenior Software EngineerCommented:
Hi Bob, sorry for not replying sooner. I feel the original question was answered as close as possible..
0
 
davidlars99Author Commented:
yes it was! thank you.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.