Solved

which p2p system can do this..

Posted on 2004-10-11
23
398 Views
Last Modified: 2010-04-10

does anyone know if any p2p protocol/system can do the following:

1) assuming say 100 dialup users, each on apx 28.8k/modems
2) assuming say 5 users on broadband apx. 512k

the 100 users each have the same copy of a movie sat star wars, encoded at 500 kilobits.

is there a way that each dialup users sends a portion of the stream to one of the broadband
users, so that the accumulated bandwidth of the dialup users is enough, that the
broadband user can watch the movie in real-time  ?

assuming each dialup users provides 1kb/sec x 100 = 100kb/s


0
Comment
Question by:apakian
  • 8
  • 7
  • 4
  • +2
23 Comments
 
LVL 3

Expert Comment

by:frieked
Comment Utility
Let me get this straight, the 100 dial up users are providing 100kb/s combined and the movie is encoded at 500kb/s.
Even if it were possible, it would take 5 seconds of xfer time to get every one second of the movie (500 / 100 = 5)
Meaning that you'd have to buffer greater than 4/5's of the movie if you wanted the xfer to finish by the time the movie ends.

0
 

Author Comment

by:apakian
Comment Utility

no,, the 100 dial up users only provide 1 kilobyte per second output each to a single 512 kilboBIT
broadband user..  i.e a total of 100kilobytes/sec from dialup to broadband..

the movie is encoded at 500kilbobits ( apx 50k/sec )

ok, say the movie is encoded at 400kilobits,, my question is if there is a protocol
that will combine seperate remote low-bandwidth users into a single high bandwidth
stream ..

0
 
LVL 1

Expert Comment

by:WanMan
Comment Utility
The problem is not so much the combination as resilience. IF one of those dial-up users drops, you need a mechanism that will account for that and degrade gracefully by spreading the load across the other dial-up users.

Suggestions are: bittorrent or perhaps variant SCTP
0
 
LVL 6

Expert Comment

by:Ferrosti
Comment Utility
This is impossible.
You assume, that every user has the whole file for the broadband user to be able to download several parts simultaneously.
Also you assume, that your movie is encoded in a streaming format. DIVX for example is not a streaming format.
0
 
LVL 3

Expert Comment

by:frieked
Comment Utility
I wouldn't go so far as to say it's impossible, improbable yes.
Basically this type of scenario would only be possible in a lab setup with some specially crafted software because like WanMan said, you'd need to rely on 100 dial-up users not losing their connection or dropping below the xfer rate for the entire length of the movie (yeah right)
0
 

Author Comment

by:apakian
Comment Utility

so is the problem latency of the notification that a call has been dropped, or
teh aggregation of the bandwidth...

for arguments sake, lets not talk about movies:

10 of my friends on dialup each have a copy of say doom3 , and they
are all running a 28.8k , using some software that reserved 30% of their
outgoing modem speed, 1kilobyte per second is used to transmit packets
to me ( im on a 512k )

user 0, sends offset 0-999,10000-10999,20000-20999 etc
user 1, send offset 1000-1999,2000-2999
...
user 9,sends offset 9000-9999
etc

if each user sustained 1 packet a second outword, what network will manage
this, so that i receive at least or close to 10kilobytes/second..

Sorry, if i'm repeating the question, and u have already understood,,
i want to hear reasons for and against..

0
 
LVL 3

Expert Comment

by:happythedog
Comment Utility
not listening the best delivery nature of tcp ip this wont work
speeds change as packets go through the wan
0
 
LVL 3

Expert Comment

by:frieked
Comment Utility
Pretty much all of the p2p networks support downloading from more than 1 host at the same time.  The file is broken into parts and downloaded at certain offsets, the faster the client the more chunks of the file you will get from that client.
0
 

Author Comment

by:apakian
Comment Utility

Yes, but their is redundancy...

say i pinged and found the location of the 100 closest users to me,, and requested the chunks
from them; i can't be certain, that he has enough bandwidth to sustain his required upstream
to me, so that the aggregated bandwidths, come to me with as little or no , delays between
the connections..

existing protocols such as gnutella do not support what i am requiring,, so does
anyone know of any other protocols: the aim a bit like RAID drives,, using many
modem users, as a single large pipe going to a few large bandwidth users....

another way to look at this is:  many clients on dial up downloading files from a high
capacity server.. turn this back to front...  many clients on dialup uploading a
shared file to a high capacity server...

how would something like this be syncronised,, so that the dialup users, send
in order and on time...

0
 
LVL 1

Expert Comment

by:WanMan
Comment Utility
As has already been said by others, the difference between what you want to do and download managers like getright, gozilla or p2p s/w like gnutella is that you are looking for real-time streaming. Even if you aren't considering a movie, you are looking to try and fill your bandwidth from an aggregation of smaller feeds.

IN theory this sounds like a sensible suggestion, by spreading the load you are spreading the risk, but in practice there are a number of hurdles to be overcome. If those of you who have already posted will allow me to plagiarise and paraphrase somewhat:

1) You need a mechanism which can ensure *timely* delivery of the packets, so either some sort of quality of service would need to be employed or you would need to build in redundancy into the feed, as well as pre-caching of a certain amount of content. Every additional bit of redundancy, whether built into the payload or by doubling up requests will eat into the theoretical maximum bandwidth.

Now, if you have sufficient redundancy built in, then you could quite happily do this with FTP! It wouldn't be secure, there would be no quality of service, but it would be doable. The simplest way would be to grab each chunk from a minimum of two people and then abort whichever one responds slowest. Crude, inelegant, but provides a partial solution.

You could add a little bit of intelligence by adding a "memory" to the receiving broadband station so that it remembers which senders are faster / more reliable and requests files from them preferentially. Because of the vagaries of the internet, setting your ageing time and the algorithm you use to determine which senders are best would be crucial to getting the most out of this approach.

2) You need a way of allowing for changes in latency, throughput and jitter.

Latency in this context is a measure of the time it takes for a packet to get from one end of the connection to the other, and could be anywhere from <1ms (on a LAN) into the seconds on dialup from a remote part of the world. Your algorithm would need to quickly take account of any drastic changes in latency.

Throughput is a way of measuring how much bandwidth is *actually* available from each client. Once again this may vary depending on their distance - in netowrk terms - from you, the business of their ISP, butterfly wings in the Amazon, you name it! In general terms, the lower your expectations in terms of throughput, the higher the percentage of the time you will achieve it.

Jitter is a measure of the variability in latency, and can actually be negative. For example, let's say that a sending station sends three packets, a, b and c. Let's also say that because of the routing algorithms in place, when it comes time to send packet "b" there is a much faster route in place. Those packets would arrive in the order b, a, c. Now, re-arranging them into the correct order takes time, and you would need to take account of this too. Of course TCP does this for you, but UDP doesn't.


In short with sufficient intelligence in both client and servers, any protocol could be pushed into service, just don't expect to get 100kbits/s out of a hundred 1kbits/s links!
0
 
LVL 1

Expert Comment

by:WanMan
Comment Utility
Apologies for the spelling mistakes.

It should, of course, be "network".

Oops!
0
Zoho SalesIQ

Hassle-free live chat software re-imagined for business growth. 2 users, always free.

 
LVL 3

Expert Comment

by:happythedog
Comment Utility
this cant be done , using tcp for this might give us a nagle algorithm issue , namly nagle causes a buffer overrun for those not familar with nagle algorithm i will post it here in a nutshell a single tcp connection can olny have 1 outstanding tcp packet to acknoledge , that means that one packet comes in on one port its got to be acklowleged before another one is sent. the relevant rfc iisnt comming to me at this moment.
youre claiming low bandwidth conects to high bandwidth, high bandwidth sends its packet out , you acknoledge, nagle issues hit when you dont have ime to ack and have to slow the speed down or risk a slowdown with a buffer overrun , frieked showed the minimum buffer you need.
If you mean trucking the links thats possibile with a switch, trunk them and stream the feed might work , but either drops or latency added by protocols with enhanced error detection would be an issue , FST might go but youd need a token ring network to use that , FST olny goes over ethernet to connect rings ,
Two FST peers can connect over High-Level Data Link Control (HDLC), Ethernet, Token
Ring, FDDI, Asynchronous Transfer Mode (ATM), or Frame Relay
Could trunk buffer as per frieked .
and go STUN AND BSTUN as per http://www.cisco.com/en/US/tech/tk827/tk369/technologies_configuration_example09186a00801241fa.shtml
but you still wont see true 100k due to latency issues , unless you trunk this wont be able to work
0
 

Author Comment

by:apakian
Comment Utility

i've been working on a p2p movie on demand system for a few months now, and your
answers have given me an indication of what is out their and what is not.

I have been testing a new topolgy which achieves what i have mentioned above,
and it works a little bit ads follows: how do you think it would work on a larger scale.

----

most architectures are either client->server, client->client or client->many clients,
and your right, syncronising, bottlenecks and timing would make low-bandwidth
aggregation impractical.

However, imagine this:

we have a fixed server, which a few hundred people connect to,

1) John connects to the server he is on a 512k connection
2) A hundred people connect to the server, on dialup
3) John wants to watch a movie, which everyone has a copy of

instead of john getting the address of the dialup users and asking each on
for fragments during the viewing; he sends packets to the server
with line status..

the server's job is to tell individual dialup users to send required packets directly
to john.

i havnt seen this topolgy in use anywhere,, does anyone know of anything
which follows this logic.





0
 

Author Comment

by:apakian
Comment Utility

yep: sorry my spelling is awful too :-)
0
 
LVL 1

Expert Comment

by:WanMan
Comment Utility
Hmmm,

The nearest analogy I can think of would be Instant messaging, but that's not a many to one application. This approach has a number of problems associated with it.

1) The server has no way of knowing that a dialup user (Alice) has in fact sent the data, or that the broadband user (Bob) has received it. Either Alice would need to send confirmation to the server, or Bob would. If it's Bob that sends the confirmation then it would have to do that for every single fragment received, from each dialup user. If you didn't need that level of confirmation then you could just send the first few confirmations and then only inform if the data doesn't turn up. Depending on your buffering there may then be time to re-acquire the information from elsewhere.

2) The server has no way of monitoring the ability of Alice to send or Bob to receive  without adding more traffic to the mix.

3) The server has no way of measuring the network parameters between Alice and Bob, only between Alice and itself and Bob and itself. To find this out would require reporting.

ON the other hand, you *could* use the server to hold a copy of the data, which it could send itself  to make up any shortfall in throughput from the dialup users. By doing this the entire system could be made more robust, since the server could pick up the slack the instant it sees a problem, source a new "Alice" and then hand off.
0
 
LVL 3

Expert Comment

by:happythedog
Comment Utility
two words Nagle Algoritm
if you go with UDP or a best effort topology drops will slow or jitter the movie unless you bufer
Its a standard networking principle that one connection per direction per port per protocol, and 512 open and ACTIVE (thats the key word )
ports will render most configs unstable , this is impossibie based on standard networking principals. theirs about 10 years of rfcs to back me  if you  TCP_SOCKETS delay to turn Nagle off , link saturation buffer overruns will result
RFC 768 - User Datagram Protocol
RFC 791 - Internet Protocol
RFC 792 - Internet Control Message Protocol
RFC 793 - Transmission Control Protocol
RFC 826 - An Ethernet Address Resolution Protocol
RFC 854 - Telnet Protocol Specification
RFC 855 - Telnet Option Specifications
RFC 856 - Telnet Binary Transmission
RFC 857 - Telnet Echo Option
RFC 858 - Telnet Suppress Go Ahead Option
RFC 859 - Telnet Status Option
RFC 860 - Telnet Timing Mark Option
RFC 861 - Telnet Extended Options: List Option
RFC 894 - Standard for the transmission of IP datagrams over Ethernet networks
RFC 896 - Congestion control in IP/TCP internetworks
RFC 959 - File Transfer Protocol
RFC 1042 - Standard for the transmission of IP datagrams over IEEE 802 networks
RFC 1072 - TCP extensions for long-delay paths
RFC 1112 - Host extensions for IP multicasting
RFC 1332 - The PPP Internet Protocol Control Protocol (IPCP)
RFC 1334 - PPP Authentication Protocols
RFC 1350 - The TFTP Protocol (Revision 2)
RFC 1441 - Introduction to version 2 of the Internet-standard Network Management Framework
its technically impossibile
and you have to consider Johns wont be the only transfer whos to say all those line status and routing tables wont drop the server
this cant be done in a production enviroment , if 10 users want a different move thats 5120 line calls and 5120 different open ports , if 2 users send information to the same computer on the same port in the same direction at the same time a collision results , this must be resolved before any further communication on that interface can occur.
for a 100 user network its 51200 ports No Known OS or system design remains stable at this level
at 1000 users you run out of ports , olny 65535 ports and 512000 connections
0
 

Author Comment

by:apakian
Comment Utility

my question is regarding streaming, not real-time broadcasting.. ( i.e on or close to  sync with
the actual source ).

the nagle algorithm causes problem in a truely serial media type,, such as voice over ip,
watching movies does not suffer:

just to clarify: of what am i saying, is TCP an issue,, the ability to hand control to a remote server
to handle the managment of packets between two peers remote to itself.

or the ability to transfer video over TCP ?
0
 
LVL 3

Expert Comment

by:happythedog
Comment Utility
the redundacny of tcp makes movies not recommended or possibile in the implememtation you describe
allow me to quote from your last post, I will then address each point in turn
"we have a fixed server, which a few hundred people connect to,

1) John connects to the server he is on a 512k connection
2) A hundred people connect to the server, on dialup
3) John wants to watch a movie, which everyone has a copy of

instead of john getting the address of the dialup users and asking each on
for fragments during the viewing; he sends packets to the server
with line status..

the server's job is to tell individual dialup users to send required packets directly
to john.

i havnt seen this topolgy in use anywhere,, does anyone know of anything
which follows this logic.
"
point  1 john connects off of a 512k CIR connection ( Committed Information Rate , means it will not drop below that level under non error conditions ) you want him to be a realtime reciever , to keep sync he cannot stop pause or otherwise influence the movie.
point 2 one hundred people connect to a server on a dial-up ( I will use US standard 53k in the us due to the FCC regulation you wont see 56k)
point 3 this data is distributed over 100 machines as described above , and you want the 100 user stream to feed johns 512k connection?
Transmission Control Protocol is the transport layer protocol , Internet Protocol is the layer 3 I presume.
do i have your requirements correct Sir/Maam
____________________________________________________________________________________________________________________
Ok assuming above is correct i will now show why it doesnt work
1. 512 users must stream , you wish to use a server ok i will give you the benefit of a server os that can take this ( remember one connection per port per protocol per direction )
2 due to dropped packets line conditiond you normally see 33.6k as a standard
3. TCP adds alot of overhead which will no doubt overrun Johns buffer
please Read RFC 791 793 826 896 1042 1072 1332 1441 before reposting
those rfcs explain why this can not be done in detail but  i shall explain here in brief
the nagle algorithm states ONE packet at a time can be out for acknoledgement.
due to tcps 3 way handshake X X+1 Y Y+1 is not possibile.
first you need a window size , too large a window size  and timeouts occur after enough timeouts tcp aborts.
too small and too litle data occurs and you end up in a nagle issue.
lets choose a window size of 3 for my sample
computer X sending  packets one two three computer Y ack 4 thats normal tcp if an error occurs as below
Computer X sending packets 1 2 3 computer Y ack 2 computer X sending three computer Y recieves 3
now the issue comes in with line access you want 100 ltcp connections happening thats 100 packets waiting on a single connection at anytime
any errors any slowdowns and the windows collapse what happens if the windows collapse the packets are retransmitted so maybe a computer drops a packet and that one has to retransmit , now remember you want all in sync they might fall out of sync if this happens combined with a whole host of wan issues causing one computer to transmit a bad packet and a loop occurs.
it doesnt matter if your watching movies using voip r anything else the rules of tcp still apply and those inherent rules make this tough to do even for one single upstream ( john ) now i doubt  you want to implement this for a single user, ad ten users that want the movie and its 1000 packets , looking for it if i find it i will post a link but Vinton Cerf has a lovley discription of this issue in his ip space protocol
0
 
LVL 3

Accepted Solution

by:
happythedog earned 350 total points
Comment Utility
http://www.ipnsig.org/techinfo.htm thats  the link , the papers here explain what you are trying to do the one about using standand internet protocol combined with RFCs 791 793 826 896 1042 1072 1332 1441 should give you a through grounding
0
 
LVL 3

Expert Comment

by:happythedog
Comment Utility
*100 should be 100 users must stream, *or
0
 

Author Comment

by:apakian
Comment Utility

ok , great wealth of info, which ive been reading thankyou.
however, in the near future i will announce a demonstration of what we have been discussing,
and possibly how im doing it..

thankyou again.
0
 
LVL 3

Expert Comment

by:happythedog
Comment Utility
apakian , you dont have an e-mail listed and i have no other contact information , i would be intrested in assisting you further
if i can be of more help to you my e-mail is weeder45@optonline.net youre quite welcome
0
 

Author Comment

by:apakian
Comment Utility

it ashod@apakian.com  , ive set it in my profile, but does not appear..

ive also got an interesting demo ive been working on tonight, ill email
you and let you know about it..
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now