How to create an HTTP Streaming Receiver

Hi,

I am working on a project which involves the implementation of an HTTPS "Events Streamer".  I need to open a streaming connection to receive async real time authorized accounts’ events.  Ideally, I need a class object that allows me to initiate the connection and receive the chunked async messages through an event handler into my main form.

Aside from the token requirement which I know how to handle because I already have two function calls already made, one is a WebGetRequest function and the other is a WebPostRequest function.  They are designed as a simple request/response pattern.  However, the Events Streaming seems to be a chunked method where chunks are sent periodically and then need to be received and sent through and event handler into the main form.  I don't quite know how to implement such a thing.

Here is the exact reference to what I am talking about:
http://developer.oanda.com/rest-live/streaming/


Refer to the Events Streaming section.
I need some sample vb.net code to get me going with this...
Very much appreciate the help...
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David Johnson, CD, MVPOwnerCommented:
one would normally have a message-id on the server that is incremented as events accumulate.  The client will have a timer that polls the server and if the message-id is greater than the last received message then retrieve the message
RobertFromSecretWeaponsAuthor Commented:
Hi David,

I don't think this is what will be going on here in this case.  I believe chunking is how this is going to go.  To get the stream started, I need to issue a Get or Post request to get it started, and messages (chunks) will arrive async,  For instance, take the heart beat message, it will arrive eveery 15 seconds by itself.  I may need to setup one of the response methods to raise an event when a message comes in.....
BigRatCommented:
My 2 cents worth, and this may not be the solution you require, but you'll need to find a class/object which allows you to periodically "poll" the connection to see if a chunk has arrived or cause an event for it. So thequestion is, is there such an object in .net, because I don't know .net. But it might be possible to put a ReadStream object onto an HTTP object(?)

It is of course dead easy to do this with node.js, since setting up a basic http/https connection is easy and one registers with "on" an event handler which gets chunks of data.
HTML5 and CSS3 Fundamentals

Build a website from the ground up by first learning the fundamentals of HTML5 and CSS3, the two popular programming languages used to present content online. HTML deals with fonts, colors, graphics, and hyperlinks, while CSS describes how HTML elements are to be displayed.

RobertFromSecretWeaponsAuthor Commented:
Hi BigRat,

Well, I am using the vb.net / framework 4.0, WebRequest, HTTPWebResponse classes.  The chunks are being received but I think they are somehow part of each received Header.  The Response.contentLength always = -1 when chunks are received.  I am single stepping through the debugger and looking at the
Dim Response As HttpWebResponse = CType(Request.GetResponse, HttpWebResponse)

I see some received headers in the Response. but I dont know where the actual data chunklets are hidden (in which method or property) of the received Response.

At this point I just need a few lines of sample code that digs into the Response object to get the chunks.  The chunks in this case a rather short.  They are little JSON response objects around 50 bytes long or so.
BigRatCommented:
The Content-Length property is -1 because it is not set in the Response header. It is not set in the response header because the transfer encoding is chunked. In fact in each chunk there is a header and in that there is a Content-Length but that applies to that chunk only.

As far as I can see I don't think you can use this class to read the chunked bits of the returned data, since the Stream built onto WebResponse looses the chunked delimiters - ie: you can't tell where one begins and one ends. I think you need a much lower level class. I have not been able to find one.

There is a very long and confusing discussion at https://social.msdn.microsoft.com/Forums/en-US/4f28d99d-9794-434b-8b78-7f9245c099c4/problems-with-httpwebrequest-and-transferencoding-chunked?forum=ncl and the gist of it is, use use NetworkStream to connect to port 80 of remote server and then send the HTTP command + headers accordingly, you can read the same NetworkStream to get the response. Just that it'd be lots work to implement various RFC standards for HTTP.

Just a couple of notes. The oanda interface doesn't require lots of headers with the request, so that should be reasonably simple. But each chunk would seem to be a complete JSON object. A chunk is NOT a TCP packet, but a sort of http packet. It has a normal header like an HTTP response and it should have a content-length header so say how long it is, a content-type to say JSON and strickly speaking, if the data is compressed, a transfer-encoding header specifying how.

If you look at the code samples at oanda.com, the javascript implementation uses JQuery, so you might find that the XMLHTTP object in MSXML does call the callback for each chunk.
RobertFromSecretWeaponsAuthor Commented:
Here is how to do it...
A whole working class.  Just substitute your own print statements instead of mine...
With thanks to my junior associate, Crazy Cat...

Imports OandaRobotTrader.Class_OandaCommon
Imports OandaRobotTrader.Class_CommonSubroutines
Imports System.Net
Imports System.Text
Imports System.IO
Imports System.Threading

Public Class Class_OandaEventOrRateStreamer

    '****************************************************************************************************
    ' Notes can be found here: http://developer.oanda.com/rest-live/streaming/
    ' Rates Streaming: Up to 2 active rate stream connections per access token.
    ' Events Streaming: Up to 5 connections per access token.
    '****************************************************************************************************
    Public Sub New(ByRef LS As Class_Logger, ByVal StreamType As Stream_Type)
        _LS = LS
        _StreamType = StreamType

        If _StreamType = Stream_Type.Events Then
            _StreamURL = String.Format("{0}accountIds={1}", Class_OandaCommon.StreamEvents, Class_OandaCommon.AccountID)
        Else
            _StreamURL = String.Format("{0}accountId={1}&instruments=EUR_USD", Class_OandaCommon.StreamRates, Class_OandaCommon.AccountID)
        End If

        _Token = Class_OandaCommon.AccessToken

        _SC = System.Windows.Forms.WindowsFormsSynchronizationContext.Current

        thEventMonitor.IsBackground = False
        thEventMonitor.Priority = Threading.ThreadPriority.Normal
        thEventMonitor.Start()
    End Sub

    Enum Stream_Type
        Rates
        Events
    End Enum

    Private _LS As Class_Logger

    Private _StreamType As Stream_Type
    Private _StreamURL As String
    Private _Token As String

    Private SR As StreamReader

    Dim ReceiverLoopEnabled As Boolean = True
    Dim WebRequestDuration As Integer
    Dim WebResponseData As String
    Dim WebResponseStatusCode As String
    Dim WebResponseStatusDescription As String

    Public Event OandaNewMessage(ByVal Message As String)
    Private DelegOandaNewEvent As New SendOrPostCallback(AddressOf EventHasArrived)
    Private _SC As System.Threading.SynchronizationContext

    Dim thEventMonitor As New Thread(New ThreadStart(AddressOf MessageMonitor))

    Dim TS As Date

    Public Sub Disconnect()
        Dim FN As String = "Disconnect" & _StreamType.ToString & "Stream"
        Try
            ReceiverLoopEnabled = False
            SR.Close()  'because SR.ReadLine() will be blocking most of the time....

        Catch ex As Exception
            _LS.PrintMsg(FN, "CATCH1:(Message):[" & ex.Message & "]")
            _LS.PrintMsg(FN, "CATCH1:(StackTrace):[" & ex.StackTrace & "]")
            _LS.PrintMsg(FN, "CATCH1:(TargetSite):[" & ex.TargetSite.ToString & "]")
        End Try
    End Sub

    Private Sub EventHasArrived(ByVal Message As String)
        RaiseEvent OandaNewMessage(Message)
    End Sub

    Private Sub MessageMonitor()
        Dim FN As String = _StreamType.ToString & "MessageMonitor"

        While (ReceiverLoopEnabled)
            _LS.PrintMsg(FN, "Has Started...")

            Dim Request As WebRequest
            Request = WebRequest.Create(_StreamURL)
            Request.ContentType = "application/json"
            Request.Headers("Authorization") = "Bearer " & _Token
            Request.Method = "GET"

            'This could throw an exception....
            Dim Response As HttpWebResponse = CType(Request.GetResponse, HttpWebResponse)

            While (ReceiverLoopEnabled)
                Try
                    '_LS.PrintMsg(FN, "Step 2...")
                    SR = New IO.StreamReader(Response.GetResponseStream())
                    While ReceiverLoopEnabled AndAlso Not SR.EndOfStream
                        '_LS.PrintMsg(FN, "Step 3...")
                        WebResponseData = SR.ReadLine()
                        '_LS.PrintMsg(FN, "Step 4...")
                        _SC.Post(DelegOandaNewEvent, WebResponseData)
                        '_LS.PrintMsg(FN, "Step 5...")
                    End While

                Catch ex As Exception
                    _LS.PrintMsg(FN, "CATCH1:(Message):[" & ex.Message & "]")
                    _LS.PrintMsg(FN, "CATCH1:(StackTrace):[" & ex.StackTrace & "]")
                    _LS.PrintMsg(FN, "CATCH1:(TargetSite):[" & ex.TargetSite.ToString & "]")
                    SendAlertMessage(Class_SendAlertEmail.AlertSeverity.ERROR, FN, "Streaming Connection Lost.", ex.Message)
                End Try

                SR.Close()

                If ReceiverLoopEnabled Then
                    Thread.Sleep(1000) 'to ensure a pause in case of reconnect failures...
                End If

            End While

        End While
    End Sub
End Class

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
RobertFromSecretWeaponsAuthor Commented:
It looks like this is the quality answer I was looking for from the beginning....
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Visual Basic.NET

From novice to tech pro — start learning today.