imarshad
asked on
Winsock Help needed....... (Part 2)
Hi all,
Please first have a look at this Question ..
https://www.experts-exchange.com/questions/21115102/Winsock-Help-needed-Part-1.html.
Idle_Mind has been kind enough to share some of his precious time and code in solving my problem........With his help I was able to connect Multiple clients on one server......Almost all the Authentication and Connection problems are sorted out......
Now I come to my second problem.......I need to send some data from my Server to the Client.......This data can be anywhere between 280 bytes to x*280 bytes where x can be any integer even 0.1 million(as an extreme case).....but usually it will be round 1-500......I load this data into a String Variable......Now I want the following information....
I know that I will need to parse that data into chunks.....I am comfortable with this but what I want to know is the max. limit of bytes that can be send in one chunk without any loss......I know that there might be different suggestions from different people
but I want to know a safe value so that my chunks dont get corrupted....
Will I need to add some delimitir at the end of each chunk so that I am sure that I have received the full chunk at the Client side
so my communication should be like this......(as i found out on some website)
Server: I'm ready to send you a file now!
Client: Ok, go ahead and send me the first chunk.
Server: Here it is!
Client: Thanks, go ahead and send me the next chunk.
Server: Here it is!
Client: Thanks, go ahead and send me the next chunk.
Server: Here it is!
...
Server: There are no more chunks!
Client: That's all the data? Thanks for the file!
Can someone help me in this regard so that the system is compatible to that code which Idle_Mind has suggested me.....
In fact I am very busy in some other part of the project. So that is why I am getting almost no time to code it by myself.....So a working example (that fits in the sample provided by Idle_Mind ) will be greatly appreciated.....
Imran Arshad
Please first have a look at this Question ..
https://www.experts-exchange.com/questions/21115102/Winsock-Help-needed-Part-1.html.
Idle_Mind has been kind enough to share some of his precious time and code in solving my problem........With his help I was able to connect Multiple clients on one server......Almost all the Authentication and Connection problems are sorted out......
Now I come to my second problem.......I need to send some data from my Server to the Client.......This data can be anywhere between 280 bytes to x*280 bytes where x can be any integer even 0.1 million(as an extreme case).....but usually it will be round 1-500......I load this data into a String Variable......Now I want the following information....
I know that I will need to parse that data into chunks.....I am comfortable with this but what I want to know is the max. limit of bytes that can be send in one chunk without any loss......I know that there might be different suggestions from different people
but I want to know a safe value so that my chunks dont get corrupted....
Will I need to add some delimitir at the end of each chunk so that I am sure that I have received the full chunk at the Client side
so my communication should be like this......(as i found out on some website)
Server: I'm ready to send you a file now!
Client: Ok, go ahead and send me the first chunk.
Server: Here it is!
Client: Thanks, go ahead and send me the next chunk.
Server: Here it is!
Client: Thanks, go ahead and send me the next chunk.
Server: Here it is!
...
Server: There are no more chunks!
Client: That's all the data? Thanks for the file!
Can someone help me in this regard so that the system is compatible to that code which Idle_Mind has suggested me.....
In fact I am very busy in some other part of the project. So that is why I am getting almost no time to code it by myself.....So a working example (that fits in the sample provided by Idle_Mind ) will be greatly appreciated.....
Imran Arshad
ASKER
>> What you can do is control the size of what you apply to the .Send method. If you call Winsock's .Send with a block
>>larger than the MTU, then TCP will slice it up itself - just don't worry about this.
So logically if I send my 200 kB String variable it should cause no problems.....as TCP will send my data in chunks of 576 bytes and at the receiver end I will receive the chunks sequentially.....and since my end of file is followed by "***" then i will know that all data has reached me.....
I will be sending data from Server as follows
Server.send "###"
server.send MyString '200 kBytes in this variable
Server.send "***"
Can it be possible that I receive "***" and I may miss some packet from the 200kB file ? or will TCP ensure that I will receive all the data from MyString and then I will receive "***" ?
But still I see people using Packet Delimitir for transfering files via Winsock......
>>Other things to consider include whether the data arrives in bursts or is instantaneously available. A way to deal with
>>bursty data do this is to create an 8K or 64K buffer and add messages into the buffer, and then transmit the buffer (less
>>the unused bytes) when there is no space for the next data block, or when some time limit expires.
What if I use the following code
Private Sub Winsock1_DataArrival(index As Integer, ByVal bytesTotal As Long)
Dim inputString As String
Winsock1.GetData inputString, vbString
inputBuffer= inputBuffer& inputString
call CheckStartEndFile( )
End Sub
Where the CheckStartEndFIle checks for "###" as a start and "***" as the end of file and writes all that is between them into the file......Is this a good approach ? Will it work every time ? Is there some better suggestions ?
or should I use the approach that I have suggested in my Question ? If yes then I need some code to fit into my existing code (from Idle_Mind)......
Imran Arshad
>>larger than the MTU, then TCP will slice it up itself - just don't worry about this.
So logically if I send my 200 kB String variable it should cause no problems.....as TCP will send my data in chunks of 576 bytes and at the receiver end I will receive the chunks sequentially.....and since my end of file is followed by "***" then i will know that all data has reached me.....
I will be sending data from Server as follows
Server.send "###"
server.send MyString '200 kBytes in this variable
Server.send "***"
Can it be possible that I receive "***" and I may miss some packet from the 200kB file ? or will TCP ensure that I will receive all the data from MyString and then I will receive "***" ?
But still I see people using Packet Delimitir for transfering files via Winsock......
>>Other things to consider include whether the data arrives in bursts or is instantaneously available. A way to deal with
>>bursty data do this is to create an 8K or 64K buffer and add messages into the buffer, and then transmit the buffer (less
>>the unused bytes) when there is no space for the next data block, or when some time limit expires.
What if I use the following code
Private Sub Winsock1_DataArrival(index
Dim inputString As String
Winsock1.GetData inputString, vbString
inputBuffer= inputBuffer& inputString
call CheckStartEndFile( )
End Sub
Where the CheckStartEndFIle checks for "###" as a start and "***" as the end of file and writes all that is between them into the file......Is this a good approach ? Will it work every time ? Is there some better suggestions ?
or should I use the approach that I have suggested in my Question ? If yes then I need some code to fit into my existing code (from Idle_Mind)......
Imran Arshad
Imran,
You need to understand that there are two different kinds of streams in a file transfer application and that those streams require very different handling.
The first type of stream is known as the control stream which handles all the handshaking. This typically involves a login validation if there is one and the exchange of header information for the filename(s) and size(s) to be transferred. It is this stream that requires a persistent buffer since this data makes no sense when it is fragmented.
The second type of stream is known as the data stream which consists of the raw data from the file(s) being transferred. A persistent buffer is not required for the data stream. The raw data will arrive in the order it was sent, just possibly not in the same chunk sizes. This doesn't matter since all you need to do is write whatever has arrived directly into the file as you receive it. If you exchange the number of bytes that are going to be sent beforehand and then track how many have acutally arrived, you will know when to expect the end of file marker without needing to search for it every time. Consequently, you don't need to temporarily store the entire data stream until you receive the end of file marker. Instead, you can leave the file open and write each chunk as it arrives.
It is actually much easier to implement a file transfer using two concurrent connections instead of only one. The control stream can be handled by one connection while the data stream is handled by the other. This is actually the way FTP transfers are implemented. Since only raw file data flows on the second connection, you don't need to worry about finding the begininning or ending markers in the stream.
Idle_Mind
You need to understand that there are two different kinds of streams in a file transfer application and that those streams require very different handling.
The first type of stream is known as the control stream which handles all the handshaking. This typically involves a login validation if there is one and the exchange of header information for the filename(s) and size(s) to be transferred. It is this stream that requires a persistent buffer since this data makes no sense when it is fragmented.
The second type of stream is known as the data stream which consists of the raw data from the file(s) being transferred. A persistent buffer is not required for the data stream. The raw data will arrive in the order it was sent, just possibly not in the same chunk sizes. This doesn't matter since all you need to do is write whatever has arrived directly into the file as you receive it. If you exchange the number of bytes that are going to be sent beforehand and then track how many have acutally arrived, you will know when to expect the end of file marker without needing to search for it every time. Consequently, you don't need to temporarily store the entire data stream until you receive the end of file marker. Instead, you can leave the file open and write each chunk as it arrives.
It is actually much easier to implement a file transfer using two concurrent connections instead of only one. The control stream can be handled by one connection while the data stream is handled by the other. This is actually the way FTP transfers are implemented. Since only raw file data flows on the second connection, you don't need to worry about finding the begininning or ending markers in the stream.
Idle_Mind
ASKER
>>You need to understand that there are two different kinds of streams in a file transfer application and that those streams
>>require very different handling.
>>It is actually much easier to implement a file transfer using two concurrent connections instead of only one. The control
>>stream can be handled by one connection while the data stream is handled by the other.
But due to my lack of in depth knowledge I have developed my application around the same lines.... i.e I use the same control to authenticate and then Send File in chunks........
I have something like this in the DataArrival Event of Server......
Case "Update"
'Check the "Client Info" and "Password" and then authenticate.....and then tell the Client I am ready to send you data
Case "NEXT"
'Send the first Chunk.......If there is no more chunks then Send EOF Marker.......
'It will be of the form "Chunk & chr(0) & Data & chr(1)"
and in the Data Arrival of Client I have
Case "SendingData"
'Initialize a buffer to receive the data and send "NEXT"
Case "Chunk"
This is a data chunk so store in the buffer allocated.....Now Send "NEXT"
Case "EOF"
Data Completed.....So save the file........
This is working for me now.....I have left quite few details but I think it will be enough to make you understand my algorithm
But according to you it is better to have a Second Winsock control so that after the authentication part is completed on one winsock only Data should be send on the other winsock.........
Can you please tell me why my technique is inferior to yours ? What are the glitches or problems that I will face if I use my technique? though it is working fine right now........
Just a side question have you used wininet.dll?
Imran Arshad
>>require very different handling.
>>It is actually much easier to implement a file transfer using two concurrent connections instead of only one. The control
>>stream can be handled by one connection while the data stream is handled by the other.
But due to my lack of in depth knowledge I have developed my application around the same lines.... i.e I use the same control to authenticate and then Send File in chunks........
I have something like this in the DataArrival Event of Server......
Case "Update"
'Check the "Client Info" and "Password" and then authenticate.....and then tell the Client I am ready to send you data
Case "NEXT"
'Send the first Chunk.......If there is no more chunks then Send EOF Marker.......
'It will be of the form "Chunk & chr(0) & Data & chr(1)"
and in the Data Arrival of Client I have
Case "SendingData"
'Initialize a buffer to receive the data and send "NEXT"
Case "Chunk"
This is a data chunk so store in the buffer allocated.....Now Send "NEXT"
Case "EOF"
Data Completed.....So save the file........
This is working for me now.....I have left quite few details but I think it will be enough to make you understand my algorithm
But according to you it is better to have a Second Winsock control so that after the authentication part is completed on one winsock only Data should be send on the other winsock.........
Can you please tell me why my technique is inferior to yours ? What are the glitches or problems that I will face if I use my technique? though it is working fine right now........
Just a side question have you used wininet.dll?
Imran Arshad
I think what Idle_Mind may have forgot to mention is that twin stream applications are usually implemented in multithreaded systems where the control thread can send a signal (unblock a Mutex) to the other thread which then responds to the request for more data. This is great in C/C++ if you're writing a high performance NT service - but very very difficult (multiple threads that is) in VB.
The wininet.dll handles a lot of async stuff for you in providing the notification events to make writing simple TCP/IP a lot easier in VB. I personally find that unless performance is a major issue that it's a lot more productive to be able to write something in VB in a tenth of the time that it would take in C++, so I use the control.
It is possible to get great performance with a single stream. The technique of prefixing the block with a "type" marker so that the receiver knows whether it's control or data is well established.
I think that now that you have it sorted out and the performance is OK then you should stick with what you have.
Cheers,
Wes
The wininet.dll handles a lot of async stuff for you in providing the notification events to make writing simple TCP/IP a lot easier in VB. I personally find that unless performance is a major issue that it's a lot more productive to be able to write something in VB in a tenth of the time that it would take in C++, so I use the control.
It is possible to get great performance with a single stream. The technique of prefixing the block with a "type" marker so that the receiver knows whether it's control or data is well established.
I think that now that you have it sorted out and the performance is OK then you should stick with what you have.
Cheers,
Wes
>> But according to you it is better to have a Second Winsock control so that after the authentication part is completed on >> one winsock only Data should be send on the other winsock.........
>> Can you please tell me why my technique is inferior to yours ? What are the glitches or problems that I will face if I use >> my technique? though it is working fine right now........
I don't think either method, single or dual winsock controls, is better than the other. They both have their advantages and disadvantages.
My main point was that you need to handle the two differenct kinds of streams, not connections differently: Control streams need a persistent buffer while data streams do not.
Idle_Mind
>> Can you please tell me why my technique is inferior to yours ? What are the glitches or problems that I will face if I use >> my technique? though it is working fine right now........
I don't think either method, single or dual winsock controls, is better than the other. They both have their advantages and disadvantages.
My main point was that you need to handle the two differenct kinds of streams, not connections differently: Control streams need a persistent buffer while data streams do not.
Idle_Mind
ASKER
Hi,
As stated in my earlier post I have developed the algorithm of sending my data over the Winsock......I tried it on Single PC as well as on a Network......It worked quite well and fast enough......But now I am testing it on the Internet and the file transfer is really slow.....I transfered a file of about 967kB and it took more then 1 hr to transfer....To check my internet connection downloading speed I also downloaded a file from Some Website of about the same size and it took about 5 min. to download a 980 kB file......SO this is highly undesireable for me......My technique is as follows....
After authentication and Verification I divide the data to be transfered into small chunks......Each of this chunk will have 10*280 bytes i.e 2800 bytes.... and each chunk is embedded with a terminating character.....When my Client receives the terminating character it requests the next chunk (This makes sure that the earlier chunk is received and then the request for next chunk is issued).....
I have tested on my Client that the next Chunk reaches after variable time......When I issue "Next" command from the client the "Chunk" (2800 bytes) reaches me sometimes in 5 sec. and sometimes in 20 Sec. (really slow)... or even more late then this.....What could be the reason ? Shouldn't the 2800 bytes reach me within 1-2 sec. ?
I have read on the internet that the file should be break into chunks and when each chunk reaches at the Client then the Next chunk should be sent.......Is this correct?
Also is 2800 bytes a good chunk size ? Should it be smaller then this or larger then this? If I change it to say 5600 bytes then will I face loss of Packets? or not?
Imran Arshad
As stated in my earlier post I have developed the algorithm of sending my data over the Winsock......I tried it on Single PC as well as on a Network......It worked quite well and fast enough......But now I am testing it on the Internet and the file transfer is really slow.....I transfered a file of about 967kB and it took more then 1 hr to transfer....To check my internet connection downloading speed I also downloaded a file from Some Website of about the same size and it took about 5 min. to download a 980 kB file......SO this is highly undesireable for me......My technique is as follows....
After authentication and Verification I divide the data to be transfered into small chunks......Each of this chunk will have 10*280 bytes i.e 2800 bytes.... and each chunk is embedded with a terminating character.....When my Client receives the terminating character it requests the next chunk (This makes sure that the earlier chunk is received and then the request for next chunk is issued).....
I have tested on my Client that the next Chunk reaches after variable time......When I issue "Next" command from the client the "Chunk" (2800 bytes) reaches me sometimes in 5 sec. and sometimes in 20 Sec. (really slow)... or even more late then this.....What could be the reason ? Shouldn't the 2800 bytes reach me within 1-2 sec. ?
I have read on the internet that the file should be break into chunks and when each chunk reaches at the Client then the Next chunk should be sent.......Is this correct?
Also is 2800 bytes a good chunk size ? Should it be smaller then this or larger then this? If I change it to say 5600 bytes then will I face loss of Packets? or not?
Imran Arshad
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
If you study my implementation of dual winsocks closely, you will see that there is no need for an ack signal at all since you get a SendComplete() event when the last chunk was successfully received. The first chunk is sent from the Connect() event and then the rest of file is sent in chunks each time SendComplete() fires. After sending that first chunk, the code never leaves the SendComplete() sub unless you receive a chat message.
Idle_Mind
Idle_Mind
ASKER
VERSION 5.00
Object = "{248DD890-BB45-11CF-9ABC- 0080C7E7B7 8D}#1.0#0" ; "MSWINSCK.OCX"
Object = "{3B7C8863-D78F-101B-B9B5- 04021C0094 02}#1.2#0" ; "RICHTX32.OCX"
Object = "{F9043C88-F6F2-101A-A3C9- 08002B2F49 FB}#1.2#0" ; "COMDLG32.OCX"
Object = "{831FDD16-0C5C-11D2-A9FC- 0000F8754D A1}#2.0#0" ; "MSCOMCTL.OCX"
I never knew that VB programs can be written like this......I think I will need quite lot of time to understand your code and then embed in my code......I will try it tomorrow...... By the way what is the advantage of using OCX controls like this and not adding them from Components Tab?
and similarly
Begin VB.CommandButton Command3
Caption = "Command3"
Height = 375
Left = 3000
TabIndex = 2
Top = 600
Width = 1335
End
What is the advantage of using it like this rather then using the standard method of adding a Command Button on the Form and changing its properties ?
I see one big advantage for me......My boss will be more impressed with the total no. of lines increased significantly (Laughing out loudly)......(He is a non IT person and don't know much about programming).........
Imran Arshad
Object = "{248DD890-BB45-11CF-9ABC-
Object = "{3B7C8863-D78F-101B-B9B5-
Object = "{F9043C88-F6F2-101A-A3C9-
Object = "{831FDD16-0C5C-11D2-A9FC-
I never knew that VB programs can be written like this......I think I will need quite lot of time to understand your code and then embed in my code......I will try it tomorrow...... By the way what is the advantage of using OCX controls like this and not adding them from Components Tab?
and similarly
Begin VB.CommandButton Command3
Caption = "Command3"
Height = 375
Left = 3000
TabIndex = 2
Top = 600
Width = 1335
End
What is the advantage of using it like this rather then using the standard method of adding a Command Button on the Form and changing its properties ?
I see one big advantage for me......My boss will be more impressed with the total no. of lines increased significantly (Laughing out loudly)......(He is a non IT person and don't know much about programming).........
Imran Arshad
That is the way VB stores your form in the form file. I did not create that code manually, VB updates all that information as you add and/or change controls to the form and change the properties of those controls. You can view your forms code by simply opening it with Notepad. =)
Follow the instructions I gave above the code and in my last post to recreate the application. Experiment with the app as it is before trying to integrate into your application. The app I just gave you has a very different architecture than what you were making before.
Here are my instructions from the last post:
To create this application, paste the code below into notepad and save it as Form2.frm. Then create a new project and add the newly created form to it. Remove the default form from the project. Be sure to go in to Project --> Properties and reset the Startup Object field. Compile the application and run two instances of the EXE.
Hit the Listen button on one of the instances. In the other instance, type in the IP shown or the local IP (127.0.0.1) and hit connect. Once you have a connection, you may chat between the applications. Now hit the Transfer button. A File Open dialog box will appear allowing you to select the file to send. Once the file has been selected, a Save As dialog box will appear for the other application. If the receiver accepts the file then transfer will begin. Chatting my continue while the file is being transferred.
Regards,
Idle_Mind
Follow the instructions I gave above the code and in my last post to recreate the application. Experiment with the app as it is before trying to integrate into your application. The app I just gave you has a very different architecture than what you were making before.
Here are my instructions from the last post:
To create this application, paste the code below into notepad and save it as Form2.frm. Then create a new project and add the newly created form to it. Remove the default form from the project. Be sure to go in to Project --> Properties and reset the Startup Object field. Compile the application and run two instances of the EXE.
Hit the Listen button on one of the instances. In the other instance, type in the IP shown or the local IP (127.0.0.1) and hit connect. Once you have a connection, you may chat between the applications. Now hit the Transfer button. A File Open dialog box will appear allowing you to select the file to send. Once the file has been selected, a Save As dialog box will appear for the other application. If the receiver accepts the file then transfer will begin. Chatting my continue while the file is being transferred.
Regards,
Idle_Mind
Actually, by build instructions are slightly out of order. After saving the code through notepad into Form1.frm (the name doesn't matter), create a new project and remove the default form. Then you can add your newly created form to the project throur Project --> Add Form --> Existing.
Idle_Mind
Idle_Mind
ASKER
Part 3 of this series.....
https://www.experts-exchange.com/questions/21343299/Winsock-Help-needed-Part-3.html
Imran
https://www.experts-exchange.com/questions/21343299/Winsock-Help-needed-Part-3.html
Imran
You can't directly control how TCP will do this with the Winsock control because it does not allow you to allow you so set socket options like e.g. Nagle Algorithm as you can in C/C++. The Nagle Algorithm is a sort of optimisation tool for TCP because it allows it to buffer up the data into blocks to get as many bytes into a packet so that you don't end up sending a packet for every single byte which would be very inefficient because of protocol overhead.
TCP has its own recovery mechanisms so corruption should in theory not happen (when you get a corrupt download it is likely not because of TCP, but because of a corrupted file on the web server), but if you want belt and braces then you can always add your own checksum at the end of your blocks.
One thing to consider is how TCP/IP itself manages its own blocks. The "MTU" defines the maximum size of a single packet, e.g. for ethernet is 1040 bytes and 576 bytes for dialup. These packets are below the TCP stream level but may be a useful indicator. You may also want to consider the size of the data blocks that you're transmitting. What you can do is control the size of what you apply to the .Send method. If you call Winsock's .Send with a block larger than the MTU, then TCP will slice it up itself - just don't worry about this.
Other things to consider include whether the data arrives in bursts or is instantaneously available. A way to deal with bursty data do this is to create an 8K or 64K buffer and add messages into the buffer, and then transmit the buffer (less the unused bytes) when there is no space for the next data block, or when some time limit expires. Think also about how long it will take to transmit each chunk - because that will affect timeouts that you code into the Server to detect dead connections - is this an issue - how long should the server wait before it realises that a client is dead? One advantage of small frequent blocks is that you're closer able to monitor network congestion more quickly and control the rate at which you send data if you so desire.