Link to home
Start Free TrialLog in
Avatar of XK8ER
XK8ERFlag for United States of America

asked on

Using loop to find low prices

Hello,
I am trying to find the fastest possible way to loop through a ConcurrentDictionary to find low prices in the market.
I am averaging 450 loops per second on a computer using 8 cores. Someone suggested to me to look into (intel xeon phi) with 60 cores.
the problem with that is that I don't know if it would work with programming languages like vb.net or c#.
in this example I am using regular For Each and also using Parallel.ForEach with a bit higher load.
I need to find a way to loop through this dictionary about 5,000 times per second with low CPU load and not just 450p/s..
if new hardware is required then I'm open for any suggestions.

Option Explicit On
Option Strict On

Imports System.Collections.Concurrent

Public Class Form1

    Public StoredStockPrices As New ConcurrentDictionary(Of Integer, StockPrices)
    Dim r As New Random
    Dim iLoops As Boolean
    Dim iTotalLoopsPM As Long

    Private Sub Form1_Closing(sender As Object, e As System.ComponentModel.CancelEventArgs) Handles MyBase.Closing
        iLoops = False
    End Sub

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        If StoredStockPrices.Count = 0 Then
            For x = 1 To 15000
                StoredStockPrices(x) = New StockPrices(r.Next, r.Next.ToString, r.Next.ToString, r.Next.ToString,
                                                       r.Next.ToString, r.Next, r.Next, True,
                                                       New TimeSpan(1, 0, 0), #07/29/2016 2:00 PM#, #07/30/2016 2:00 PM#, r.Next, r.Next,
                                                       r.Next, r.Next, r.Next)
            Next
        End If

        Debug.Print(StoredStockPrices(1).Name.ToString & "|" & StoredStockPrices.Count)
    End Sub

    Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
        Call StartLoops()
    End Sub

    Public Sub StartLoops()
        iLoops = True
        Dim iThreadLoops As New Threading.Thread(New Threading.ThreadStart(AddressOf threadLoops))
        iThreadLoops.Start()
    End Sub

    Private Sub threadLoops()
        If StoredStockPrices.Count > 0 Then
            Do While iLoops = True

                Dim copyStoredStockPrices As New ConcurrentDictionary(Of Integer, StockPrices)
                copyStoredStockPrices = StoredStockPrices

                '360 p/s with 50% CPU usage
                Threading.Tasks.Parallel.ForEach(copyStoredStockPrices.Values.Reverse, Sub(myStockPrices)
                                                                                           Call CheckLowPrices(myStockPrices)
                                                                                       End Sub)



                '450 p/s with 20% CPU usage
                'For Each myStockPrices In copyStoredStockPrices.Values.Reverse
                '    Call CheckLowPrices(myStockPrices)
                'Next

                iTotalLoopsPM += 1
                copyStoredStockPrices = Nothing

                Threading.Thread.Sleep(1)
            Loop
        End If
    End Sub

    Private Sub CheckLowPrices(myStockPrices As StockPrices)
        If myStockPrices.StrikePrice < 10 AndAlso myStockPrices.Index = 10 Then
            If myStockPrices.Sequence < 500 AndAlso myStockPrices.Index < 20 Then
                'submit order
            End If
        End If
    End Sub

    Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
        Dim PerSec As Double = (iTotalLoopsPM / Date.UtcNow.Second)
        Label1.Text = Math.Round(PerSec, 0) & "p/s"
        Label3.Text = Math.Round(iTotalLoopsPM, 0) & "p/m"

        If Date.UtcNow.Second = 0 Then
            iTotalLoopsPM = 0
        End If

        Label2.Text = Date.UtcNow.ToString

        'Stock prices are always changing, hundreds of times per second.
        If StoredStockPrices.Count > 0 Then
            For Each eachRecord In StoredStockPrices
                StoredStockPrices(eachRecord.Key) = New StockPrices(r.Next, r.Next.ToString, r.Next.ToString, r.Next.ToString,
                                                       r.Next.ToString, r.Next, r.Next, True,
                                                       New TimeSpan(1, 0, 0), #07/29/2016 2:00 PM#, #07/30/2016 2:00 PM#, r.Next, r.Next,
                                                       r.Next, r.Next, r.Next)
            Next
        End If
    End Sub
End Class

Open in new window



Public Class StockPrices
    Private m_Index As Integer
    Private m_Symbol As String
    Private m_Name As String
    Private m_ContractStatus As String
    Private m_UnderlyingSymbol As String
    Private m_UnderlyingSettledPrice As Double
    Private m_StrikePrice As Double
    Private m_HasTraded As Boolean
    Private m_LifeSpan As TimeSpan
    Private m_OpensAt As DateTime
    Private m_ClosesAt As DateTime
    Private m_CeilingPrice As Double
    Private m_FloorPrice As Double
    Private m_Latency As Long
    Private m_Sequence As Integer
    Private m_TickMath As Double

    Public Sub New(ByVal iIndex As Integer, ByVal iSymbol As String, ByVal iName As String, ByVal iContractStatus As String,
                   ByVal iUnderlyingSymbol As String, ByVal iUnderlyingSettledPrice As Double, ByVal iStrikePrice As Double,
                   ByVal iHasTraded As Boolean, ByVal iLifeSpan As TimeSpan, ByVal iOpensAt As DateTime, ByVal iClosesAt As DateTime,
                   ByVal iCeilingPrice As Double, ByVal iFloorPrice As Double, ByVal iLatency As Long, ByVal iSequence As Integer,
                   ByVal iTickMath As Double)
        Me.m_Index = iIndex
        Me.m_Symbol = iSymbol
        Me.m_Name = iName
        Me.m_ContractStatus = iContractStatus
        Me.m_UnderlyingSymbol = iUnderlyingSymbol
        Me.m_UnderlyingSettledPrice = iUnderlyingSettledPrice
        Me.m_StrikePrice = iStrikePrice
        Me.m_HasTraded = iHasTraded
        Me.m_LifeSpan = iLifeSpan
        Me.m_OpensAt = iOpensAt
        Me.m_ClosesAt = iClosesAt
        Me.m_CeilingPrice = iCeilingPrice
        Me.m_FloorPrice = iFloorPrice
        Me.m_Latency = iLatency
        Me.m_Sequence = iSequence
        Me.m_TickMath = iTickMath
    End Sub

    Public Property Index() As Integer
        Get
            Return m_Index
        End Get
        Set
            m_Index = Value
        End Set
    End Property

    Public Property Symbol() As String
        Get
            Return m_Symbol
        End Get
        Set
            m_Symbol = Value
        End Set
    End Property

    Public Property Name() As String
        Get
            Return m_Name
        End Get
        Set
            m_Name = Value
        End Set
    End Property

    Public Property ContractStatus() As String
        Get
            Return m_ContractStatus
        End Get
        Set
            m_ContractStatus = Value
        End Set
    End Property

    Public Property UnderlyingSymbol() As String
        Get
            Return m_UnderlyingSymbol
        End Get
        Set
            m_UnderlyingSymbol = Value
        End Set
    End Property

    Public Property UnderlyingSettledPrice() As Double
        Get
            Return m_UnderlyingSettledPrice
        End Get
        Set
            m_UnderlyingSettledPrice = Value
        End Set
    End Property

    Public Property StrikePrice() As Double
        Get
            Return m_StrikePrice
        End Get
        Set
            m_StrikePrice = Value
        End Set
    End Property

    Public Property LifeSpan() As TimeSpan
        Get
            Return m_LifeSpan
        End Get
        Set
            m_LifeSpan = Value
        End Set
    End Property

    Public Property HasTraded() As Boolean
        Get
            Return m_HasTraded
        End Get
        Set
            m_HasTraded = Value
        End Set
    End Property

    Public Property OpensAt() As DateTime
        Get
            Return m_OpensAt
        End Get
        Set
            m_OpensAt = Value
        End Set
    End Property

    Public Property ClosesAt() As DateTime
        Get
            Return m_ClosesAt
        End Get
        Set
            m_ClosesAt = Value
        End Set
    End Property

    Public Property CeilingPrice() As Double
        Get
            Return m_CeilingPrice
        End Get
        Set
            m_CeilingPrice = Value
        End Set
    End Property

    Public Property FloorPrice() As Double
        Get
            Return m_FloorPrice
        End Get
        Set
            m_FloorPrice = Value
        End Set
    End Property

    Public Property Latency() As Long
        Get
            Return m_Latency
        End Get
        Set
            m_Latency = Value
        End Set
    End Property

    Public Property Sequence() As Integer
        Get
            Return m_Sequence
        End Get
        Set
            m_Sequence = Value
        End Set
    End Property

    Public Property TickMath() As Double
        Get
            Return m_TickMath
        End Get
        Set
            m_TickMath = Value
        End Set
    End Property
End Class

Open in new window

Avatar of David Johnson, CD
David Johnson, CD
Flag of Canada image

have you run your code through a profiler and found any bottlenecks?
Avatar of XK8ER

ASKER

yes I have used profiler but the issue is not bottlenecks is speed
ASKER CERTIFIED SOLUTION
Avatar of Guy Hengel [angelIII / a3]
Guy Hengel [angelIII / a3]
Flag of Luxembourg image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of XK8ER

ASKER

there are two connections to two different market data feed.
the function CheckLowPrices is just something very similar to what we have in place.

example:
data feed #1 says EUR/USD contract 1.1166 is selling for $50
data feed #2 says the underlying price for EUR/USD is at 1.11652

if we only check one time it could work yes but sometimes data feed #1 doesn't move for several second so if its a good price according to data feed #2 then we miss out the window to submit a new order, and when market data #1 moves again its already not a good price