which p2p system can do this..
does anyone know if any p2p protocol/system can do the following:
1) assuming say 100 dialup users, each on apx 28.8k/modems
2) assuming say 5 users on broadband apx. 512k
the 100 users each have the same copy of a movie sat star wars, encoded at 500 kilobits.
is there a way that each dialup users sends a portion of the stream to one of the broadband
users, so that the accumulated bandwidth of the dialup users is enough, that the
broadband user can watch the movie in real-time ?
assuming each dialup users provides 1kb/sec x 100 = 100kb/s
ASKER
no,, the 100 dial up users only provide 1 kilobyte per second output each to a single 512 kilboBIT
broadband user.. i.e a total of 100kilobytes/sec from dialup to broadband..
the movie is encoded at 500kilbobits ( apx 50k/sec )
ok, say the movie is encoded at 400kilobits,, my question is if there is a protocol
that will combine seperate remote low-bandwidth users into a single high bandwidth
stream ..
The problem is not so much the combination as resilience. IF one of those dial-up users drops, you need a mechanism that will account for that and degrade gracefully by spreading the load across the other dial-up users.
Suggestions are: bittorrent or perhaps variant SCTP
Suggestions are: bittorrent or perhaps variant SCTP
This is impossible.
You assume, that every user has the whole file for the broadband user to be able to download several parts simultaneously.
Also you assume, that your movie is encoded in a streaming format. DIVX for example is not a streaming format.
You assume, that every user has the whole file for the broadband user to be able to download several parts simultaneously.
Also you assume, that your movie is encoded in a streaming format. DIVX for example is not a streaming format.
I wouldn't go so far as to say it's impossible, improbable yes.
Basically this type of scenario would only be possible in a lab setup with some specially crafted software because like WanMan said, you'd need to rely on 100 dial-up users not losing their connection or dropping below the xfer rate for the entire length of the movie (yeah right)
Basically this type of scenario would only be possible in a lab setup with some specially crafted software because like WanMan said, you'd need to rely on 100 dial-up users not losing their connection or dropping below the xfer rate for the entire length of the movie (yeah right)
ASKER
so is the problem latency of the notification that a call has been dropped, or
teh aggregation of the bandwidth...
for arguments sake, lets not talk about movies:
10 of my friends on dialup each have a copy of say doom3 , and they
are all running a 28.8k , using some software that reserved 30% of their
outgoing modem speed, 1kilobyte per second is used to transmit packets
to me ( im on a 512k )
user 0, sends offset 0-999,10000-10999,20000-20
user 1, send offset 1000-1999,2000-2999
...
user 9,sends offset 9000-9999
etc
if each user sustained 1 packet a second outword, what network will manage
this, so that i receive at least or close to 10kilobytes/second..
Sorry, if i'm repeating the question, and u have already understood,,
i want to hear reasons for and against..
not listening the best delivery nature of tcp ip this wont work
speeds change as packets go through the wan
speeds change as packets go through the wan
Pretty much all of the p2p networks support downloading from more than 1 host at the same time. The file is broken into parts and downloaded at certain offsets, the faster the client the more chunks of the file you will get from that client.
ASKER
Yes, but their is redundancy...
say i pinged and found the location of the 100 closest users to me,, and requested the chunks
from them; i can't be certain, that he has enough bandwidth to sustain his required upstream
to me, so that the aggregated bandwidths, come to me with as little or no , delays between
the connections..
existing protocols such as gnutella do not support what i am requiring,, so does
anyone know of any other protocols: the aim a bit like RAID drives,, using many
modem users, as a single large pipe going to a few large bandwidth users....
another way to look at this is: many clients on dial up downloading files from a high
capacity server.. turn this back to front... many clients on dialup uploading a
shared file to a high capacity server...
how would something like this be syncronised,, so that the dialup users, send
in order and on time...
As has already been said by others, the difference between what you want to do and download managers like getright, gozilla or p2p s/w like gnutella is that you are looking for real-time streaming. Even if you aren't considering a movie, you are looking to try and fill your bandwidth from an aggregation of smaller feeds.
IN theory this sounds like a sensible suggestion, by spreading the load you are spreading the risk, but in practice there are a number of hurdles to be overcome. If those of you who have already posted will allow me to plagiarise and paraphrase somewhat:
1) You need a mechanism which can ensure *timely* delivery of the packets, so either some sort of quality of service would need to be employed or you would need to build in redundancy into the feed, as well as pre-caching of a certain amount of content. Every additional bit of redundancy, whether built into the payload or by doubling up requests will eat into the theoretical maximum bandwidth.
Now, if you have sufficient redundancy built in, then you could quite happily do this with FTP! It wouldn't be secure, there would be no quality of service, but it would be doable. The simplest way would be to grab each chunk from a minimum of two people and then abort whichever one responds slowest. Crude, inelegant, but provides a partial solution.
You could add a little bit of intelligence by adding a "memory" to the receiving broadband station so that it remembers which senders are faster / more reliable and requests files from them preferentially. Because of the vagaries of the internet, setting your ageing time and the algorithm you use to determine which senders are best would be crucial to getting the most out of this approach.
2) You need a way of allowing for changes in latency, throughput and jitter.
Latency in this context is a measure of the time it takes for a packet to get from one end of the connection to the other, and could be anywhere from <1ms (on a LAN) into the seconds on dialup from a remote part of the world. Your algorithm would need to quickly take account of any drastic changes in latency.
Throughput is a way of measuring how much bandwidth is *actually* available from each client. Once again this may vary depending on their distance - in netowrk terms - from you, the business of their ISP, butterfly wings in the Amazon, you name it! In general terms, the lower your expectations in terms of throughput, the higher the percentage of the time you will achieve it.
Jitter is a measure of the variability in latency, and can actually be negative. For example, let's say that a sending station sends three packets, a, b and c. Let's also say that because of the routing algorithms in place, when it comes time to send packet "b" there is a much faster route in place. Those packets would arrive in the order b, a, c. Now, re-arranging them into the correct order takes time, and you would need to take account of this too. Of course TCP does this for you, but UDP doesn't.
In short with sufficient intelligence in both client and servers, any protocol could be pushed into service, just don't expect to get 100kbits/s out of a hundred 1kbits/s links!
IN theory this sounds like a sensible suggestion, by spreading the load you are spreading the risk, but in practice there are a number of hurdles to be overcome. If those of you who have already posted will allow me to plagiarise and paraphrase somewhat:
1) You need a mechanism which can ensure *timely* delivery of the packets, so either some sort of quality of service would need to be employed or you would need to build in redundancy into the feed, as well as pre-caching of a certain amount of content. Every additional bit of redundancy, whether built into the payload or by doubling up requests will eat into the theoretical maximum bandwidth.
Now, if you have sufficient redundancy built in, then you could quite happily do this with FTP! It wouldn't be secure, there would be no quality of service, but it would be doable. The simplest way would be to grab each chunk from a minimum of two people and then abort whichever one responds slowest. Crude, inelegant, but provides a partial solution.
You could add a little bit of intelligence by adding a "memory" to the receiving broadband station so that it remembers which senders are faster / more reliable and requests files from them preferentially. Because of the vagaries of the internet, setting your ageing time and the algorithm you use to determine which senders are best would be crucial to getting the most out of this approach.
2) You need a way of allowing for changes in latency, throughput and jitter.
Latency in this context is a measure of the time it takes for a packet to get from one end of the connection to the other, and could be anywhere from <1ms (on a LAN) into the seconds on dialup from a remote part of the world. Your algorithm would need to quickly take account of any drastic changes in latency.
Throughput is a way of measuring how much bandwidth is *actually* available from each client. Once again this may vary depending on their distance - in netowrk terms - from you, the business of their ISP, butterfly wings in the Amazon, you name it! In general terms, the lower your expectations in terms of throughput, the higher the percentage of the time you will achieve it.
Jitter is a measure of the variability in latency, and can actually be negative. For example, let's say that a sending station sends three packets, a, b and c. Let's also say that because of the routing algorithms in place, when it comes time to send packet "b" there is a much faster route in place. Those packets would arrive in the order b, a, c. Now, re-arranging them into the correct order takes time, and you would need to take account of this too. Of course TCP does this for you, but UDP doesn't.
In short with sufficient intelligence in both client and servers, any protocol could be pushed into service, just don't expect to get 100kbits/s out of a hundred 1kbits/s links!
Apologies for the spelling mistakes.
It should, of course, be "network".
Oops!
It should, of course, be "network".
Oops!
this cant be done , using tcp for this might give us a nagle algorithm issue , namly nagle causes a buffer overrun for those not familar with nagle algorithm i will post it here in a nutshell a single tcp connection can olny have 1 outstanding tcp packet to acknoledge , that means that one packet comes in on one port its got to be acklowleged before another one is sent. the relevant rfc iisnt comming to me at this moment.
youre claiming low bandwidth conects to high bandwidth, high bandwidth sends its packet out , you acknoledge, nagle issues hit when you dont have ime to ack and have to slow the speed down or risk a slowdown with a buffer overrun , frieked showed the minimum buffer you need.
If you mean trucking the links thats possibile with a switch, trunk them and stream the feed might work , but either drops or latency added by protocols with enhanced error detection would be an issue , FST might go but youd need a token ring network to use that , FST olny goes over ethernet to connect rings ,
Two FST peers can connect over High-Level Data Link Control (HDLC), Ethernet, Token
Ring, FDDI, Asynchronous Transfer Mode (ATM), or Frame Relay
Could trunk buffer as per frieked .
and go STUN AND BSTUN as per http://www.cisco.com/en/US/tech/tk827/tk369/technologies_configuration_example09186a00801241fa.shtml
but you still wont see true 100k due to latency issues , unless you trunk this wont be able to work
youre claiming low bandwidth conects to high bandwidth, high bandwidth sends its packet out , you acknoledge, nagle issues hit when you dont have ime to ack and have to slow the speed down or risk a slowdown with a buffer overrun , frieked showed the minimum buffer you need.
If you mean trucking the links thats possibile with a switch, trunk them and stream the feed might work , but either drops or latency added by protocols with enhanced error detection would be an issue , FST might go but youd need a token ring network to use that , FST olny goes over ethernet to connect rings ,
Two FST peers can connect over High-Level Data Link Control (HDLC), Ethernet, Token
Ring, FDDI, Asynchronous Transfer Mode (ATM), or Frame Relay
Could trunk buffer as per frieked .
and go STUN AND BSTUN as per http://www.cisco.com/en/US/tech/tk827/tk369/technologies_configuration_example09186a00801241fa.shtml
but you still wont see true 100k due to latency issues , unless you trunk this wont be able to work
ASKER
i've been working on a p2p movie on demand system for a few months now, and your
answers have given me an indication of what is out their and what is not.
I have been testing a new topolgy which achieves what i have mentioned above,
and it works a little bit ads follows: how do you think it would work on a larger scale.
----
most architectures are either client->server, client->client or client->many clients,
and your right, syncronising, bottlenecks and timing would make low-bandwidth
aggregation impractical.
However, imagine this:
we have a fixed server, which a few hundred people connect to,
1) John connects to the server he is on a 512k connection
2) A hundred people connect to the server, on dialup
3) John wants to watch a movie, which everyone has a copy of
instead of john getting the address of the dialup users and asking each on
for fragments during the viewing; he sends packets to the server
with line status..
the server's job is to tell individual dialup users to send required packets directly
to john.
i havnt seen this topolgy in use anywhere,, does anyone know of anything
which follows this logic.
ASKER
yep: sorry my spelling is awful too :-)
Hmmm,
The nearest analogy I can think of would be Instant messaging, but that's not a many to one application. This approach has a number of problems associated with it.
1) The server has no way of knowing that a dialup user (Alice) has in fact sent the data, or that the broadband user (Bob) has received it. Either Alice would need to send confirmation to the server, or Bob would. If it's Bob that sends the confirmation then it would have to do that for every single fragment received, from each dialup user. If you didn't need that level of confirmation then you could just send the first few confirmations and then only inform if the data doesn't turn up. Depending on your buffering there may then be time to re-acquire the information from elsewhere.
2) The server has no way of monitoring the ability of Alice to send or Bob to receive without adding more traffic to the mix.
3) The server has no way of measuring the network parameters between Alice and Bob, only between Alice and itself and Bob and itself. To find this out would require reporting.
ON the other hand, you *could* use the server to hold a copy of the data, which it could send itself to make up any shortfall in throughput from the dialup users. By doing this the entire system could be made more robust, since the server could pick up the slack the instant it sees a problem, source a new "Alice" and then hand off.
The nearest analogy I can think of would be Instant messaging, but that's not a many to one application. This approach has a number of problems associated with it.
1) The server has no way of knowing that a dialup user (Alice) has in fact sent the data, or that the broadband user (Bob) has received it. Either Alice would need to send confirmation to the server, or Bob would. If it's Bob that sends the confirmation then it would have to do that for every single fragment received, from each dialup user. If you didn't need that level of confirmation then you could just send the first few confirmations and then only inform if the data doesn't turn up. Depending on your buffering there may then be time to re-acquire the information from elsewhere.
2) The server has no way of monitoring the ability of Alice to send or Bob to receive without adding more traffic to the mix.
3) The server has no way of measuring the network parameters between Alice and Bob, only between Alice and itself and Bob and itself. To find this out would require reporting.
ON the other hand, you *could* use the server to hold a copy of the data, which it could send itself to make up any shortfall in throughput from the dialup users. By doing this the entire system could be made more robust, since the server could pick up the slack the instant it sees a problem, source a new "Alice" and then hand off.
two words Nagle Algoritm
if you go with UDP or a best effort topology drops will slow or jitter the movie unless you bufer
Its a standard networking principle that one connection per direction per port per protocol, and 512 open and ACTIVE (thats the key word )
ports will render most configs unstable , this is impossibie based on standard networking principals. theirs about 10 years of rfcs to back me if you TCP_SOCKETS delay to turn Nagle off , link saturation buffer overruns will result
RFC 768 - User Datagram Protocol
RFC 791 - Internet Protocol
RFC 792 - Internet Control Message Protocol
RFC 793 - Transmission Control Protocol
RFC 826 - An Ethernet Address Resolution Protocol
RFC 854 - Telnet Protocol Specification
RFC 855 - Telnet Option Specifications
RFC 856 - Telnet Binary Transmission
RFC 857 - Telnet Echo Option
RFC 858 - Telnet Suppress Go Ahead Option
RFC 859 - Telnet Status Option
RFC 860 - Telnet Timing Mark Option
RFC 861 - Telnet Extended Options: List Option
RFC 894 - Standard for the transmission of IP datagrams over Ethernet networks
RFC 896 - Congestion control in IP/TCP internetworks
RFC 959 - File Transfer Protocol
RFC 1042 - Standard for the transmission of IP datagrams over IEEE 802 networks
RFC 1072 - TCP extensions for long-delay paths
RFC 1112 - Host extensions for IP multicasting
RFC 1332 - The PPP Internet Protocol Control Protocol (IPCP)
RFC 1334 - PPP Authentication Protocols
RFC 1350 - The TFTP Protocol (Revision 2)
RFC 1441 - Introduction to version 2 of the Internet-standard Network Management Framework
its technically impossibile
and you have to consider Johns wont be the only transfer whos to say all those line status and routing tables wont drop the server
this cant be done in a production enviroment , if 10 users want a different move thats 5120 line calls and 5120 different open ports , if 2 users send information to the same computer on the same port in the same direction at the same time a collision results , this must be resolved before any further communication on that interface can occur.
for a 100 user network its 51200 ports No Known OS or system design remains stable at this level
at 1000 users you run out of ports , olny 65535 ports and 512000 connections
if you go with UDP or a best effort topology drops will slow or jitter the movie unless you bufer
Its a standard networking principle that one connection per direction per port per protocol, and 512 open and ACTIVE (thats the key word )
ports will render most configs unstable , this is impossibie based on standard networking principals. theirs about 10 years of rfcs to back me if you TCP_SOCKETS delay to turn Nagle off , link saturation buffer overruns will result
RFC 768 - User Datagram Protocol
RFC 791 - Internet Protocol
RFC 792 - Internet Control Message Protocol
RFC 793 - Transmission Control Protocol
RFC 826 - An Ethernet Address Resolution Protocol
RFC 854 - Telnet Protocol Specification
RFC 855 - Telnet Option Specifications
RFC 856 - Telnet Binary Transmission
RFC 857 - Telnet Echo Option
RFC 858 - Telnet Suppress Go Ahead Option
RFC 859 - Telnet Status Option
RFC 860 - Telnet Timing Mark Option
RFC 861 - Telnet Extended Options: List Option
RFC 894 - Standard for the transmission of IP datagrams over Ethernet networks
RFC 896 - Congestion control in IP/TCP internetworks
RFC 959 - File Transfer Protocol
RFC 1042 - Standard for the transmission of IP datagrams over IEEE 802 networks
RFC 1072 - TCP extensions for long-delay paths
RFC 1112 - Host extensions for IP multicasting
RFC 1332 - The PPP Internet Protocol Control Protocol (IPCP)
RFC 1334 - PPP Authentication Protocols
RFC 1350 - The TFTP Protocol (Revision 2)
RFC 1441 - Introduction to version 2 of the Internet-standard Network Management Framework
its technically impossibile
and you have to consider Johns wont be the only transfer whos to say all those line status and routing tables wont drop the server
this cant be done in a production enviroment , if 10 users want a different move thats 5120 line calls and 5120 different open ports , if 2 users send information to the same computer on the same port in the same direction at the same time a collision results , this must be resolved before any further communication on that interface can occur.
for a 100 user network its 51200 ports No Known OS or system design remains stable at this level
at 1000 users you run out of ports , olny 65535 ports and 512000 connections
ASKER
my question is regarding streaming, not real-time broadcasting.. ( i.e on or close to sync with
the actual source ).
the nagle algorithm causes problem in a truely serial media type,, such as voice over ip,
watching movies does not suffer:
just to clarify: of what am i saying, is TCP an issue,, the ability to hand control to a remote server
to handle the managment of packets between two peers remote to itself.
or the ability to transfer video over TCP ?
the redundacny of tcp makes movies not recommended or possibile in the implememtation you describe
allow me to quote from your last post, I will then address each point in turn
"we have a fixed server, which a few hundred people connect to,
1) John connects to the server he is on a 512k connection
2) A hundred people connect to the server, on dialup
3) John wants to watch a movie, which everyone has a copy of
instead of john getting the address of the dialup users and asking each on
for fragments during the viewing; he sends packets to the server
with line status..
the server's job is to tell individual dialup users to send required packets directly
to john.
i havnt seen this topolgy in use anywhere,, does anyone know of anything
which follows this logic.
"
point 1 john connects off of a 512k CIR connection ( Committed Information Rate , means it will not drop below that level under non error conditions ) you want him to be a realtime reciever , to keep sync he cannot stop pause or otherwise influence the movie.
point 2 one hundred people connect to a server on a dial-up ( I will use US standard 53k in the us due to the FCC regulation you wont see 56k)
point 3 this data is distributed over 100 machines as described above , and you want the 100 user stream to feed johns 512k connection?
Transmission Control Protocol is the transport layer protocol , Internet Protocol is the layer 3 I presume.
do i have your requirements correct Sir/Maam
__________________________
Ok assuming above is correct i will now show why it doesnt work
1. 512 users must stream , you wish to use a server ok i will give you the benefit of a server os that can take this ( remember one connection per port per protocol per direction )
2 due to dropped packets line conditiond you normally see 33.6k as a standard
3. TCP adds alot of overhead which will no doubt overrun Johns buffer
please Read RFC 791 793 826 896 1042 1072 1332 1441 before reposting
those rfcs explain why this can not be done in detail but i shall explain here in brief
the nagle algorithm states ONE packet at a time can be out for acknoledgement.
due to tcps 3 way handshake X X+1 Y Y+1 is not possibile.
first you need a window size , too large a window size and timeouts occur after enough timeouts tcp aborts.
too small and too litle data occurs and you end up in a nagle issue.
lets choose a window size of 3 for my sample
computer X sending packets one two three computer Y ack 4 thats normal tcp if an error occurs as below
Computer X sending packets 1 2 3 computer Y ack 2 computer X sending three computer Y recieves 3
now the issue comes in with line access you want 100 ltcp connections happening thats 100 packets waiting on a single connection at anytime
any errors any slowdowns and the windows collapse what happens if the windows collapse the packets are retransmitted so maybe a computer drops a packet and that one has to retransmit , now remember you want all in sync they might fall out of sync if this happens combined with a whole host of wan issues causing one computer to transmit a bad packet and a loop occurs.
it doesnt matter if your watching movies using voip r anything else the rules of tcp still apply and those inherent rules make this tough to do even for one single upstream ( john ) now i doubt you want to implement this for a single user, ad ten users that want the movie and its 1000 packets , looking for it if i find it i will post a link but Vinton Cerf has a lovley discription of this issue in his ip space protocol
allow me to quote from your last post, I will then address each point in turn
"we have a fixed server, which a few hundred people connect to,
1) John connects to the server he is on a 512k connection
2) A hundred people connect to the server, on dialup
3) John wants to watch a movie, which everyone has a copy of
instead of john getting the address of the dialup users and asking each on
for fragments during the viewing; he sends packets to the server
with line status..
the server's job is to tell individual dialup users to send required packets directly
to john.
i havnt seen this topolgy in use anywhere,, does anyone know of anything
which follows this logic.
"
point 1 john connects off of a 512k CIR connection ( Committed Information Rate , means it will not drop below that level under non error conditions ) you want him to be a realtime reciever , to keep sync he cannot stop pause or otherwise influence the movie.
point 2 one hundred people connect to a server on a dial-up ( I will use US standard 53k in the us due to the FCC regulation you wont see 56k)
point 3 this data is distributed over 100 machines as described above , and you want the 100 user stream to feed johns 512k connection?
Transmission Control Protocol is the transport layer protocol , Internet Protocol is the layer 3 I presume.
do i have your requirements correct Sir/Maam
__________________________
Ok assuming above is correct i will now show why it doesnt work
1. 512 users must stream , you wish to use a server ok i will give you the benefit of a server os that can take this ( remember one connection per port per protocol per direction )
2 due to dropped packets line conditiond you normally see 33.6k as a standard
3. TCP adds alot of overhead which will no doubt overrun Johns buffer
please Read RFC 791 793 826 896 1042 1072 1332 1441 before reposting
those rfcs explain why this can not be done in detail but i shall explain here in brief
the nagle algorithm states ONE packet at a time can be out for acknoledgement.
due to tcps 3 way handshake X X+1 Y Y+1 is not possibile.
first you need a window size , too large a window size and timeouts occur after enough timeouts tcp aborts.
too small and too litle data occurs and you end up in a nagle issue.
lets choose a window size of 3 for my sample
computer X sending packets one two three computer Y ack 4 thats normal tcp if an error occurs as below
Computer X sending packets 1 2 3 computer Y ack 2 computer X sending three computer Y recieves 3
now the issue comes in with line access you want 100 ltcp connections happening thats 100 packets waiting on a single connection at anytime
any errors any slowdowns and the windows collapse what happens if the windows collapse the packets are retransmitted so maybe a computer drops a packet and that one has to retransmit , now remember you want all in sync they might fall out of sync if this happens combined with a whole host of wan issues causing one computer to transmit a bad packet and a loop occurs.
it doesnt matter if your watching movies using voip r anything else the rules of tcp still apply and those inherent rules make this tough to do even for one single upstream ( john ) now i doubt you want to implement this for a single user, ad ten users that want the movie and its 1000 packets , looking for it if i find it i will post a link but Vinton Cerf has a lovley discription of this issue in his ip space protocol
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
*100 should be 100 users must stream, *or
ASKER
ok , great wealth of info, which ive been reading thankyou.
however, in the near future i will announce a demonstration of what we have been discussing,
and possibly how im doing it..
thankyou again.
apakian , you dont have an e-mail listed and i have no other contact information , i would be intrested in assisting you further
if i can be of more help to you my e-mail is weeder45@optonline.net youre quite welcome
if i can be of more help to you my e-mail is weeder45@optonline.net youre quite welcome
ASKER
it ashod@apakian.com , ive set it in my profile, but does not appear..
ive also got an interesting demo ive been working on tonight, ill email
you and let you know about it..
Even if it were possible, it would take 5 seconds of xfer time to get every one second of the movie (500 / 100 = 5)
Meaning that you'd have to buffer greater than 4/5's of the movie if you wanted the xfer to finish by the time the movie ends.