Solved

Using loop to find low prices

Posted on 2016-07-30
5
75 Views
Last Modified: 2016-08-09
Hello,
I am trying to find the fastest possible way to loop through a ConcurrentDictionary to find low prices in the market.
I am averaging 450 loops per second on a computer using 8 cores. Someone suggested to me to look into (intel xeon phi) with 60 cores.
the problem with that is that I don't know if it would work with programming languages like vb.net or c#.
in this example I am using regular For Each and also using Parallel.ForEach with a bit higher load.
I need to find a way to loop through this dictionary about 5,000 times per second with low CPU load and not just 450p/s..
if new hardware is required then I'm open for any suggestions.

Option Explicit On
Option Strict On

Imports System.Collections.Concurrent

Public Class Form1

    Public StoredStockPrices As New ConcurrentDictionary(Of Integer, StockPrices)
    Dim r As New Random
    Dim iLoops As Boolean
    Dim iTotalLoopsPM As Long

    Private Sub Form1_Closing(sender As Object, e As System.ComponentModel.CancelEventArgs) Handles MyBase.Closing
        iLoops = False
    End Sub

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        If StoredStockPrices.Count = 0 Then
            For x = 1 To 15000
                StoredStockPrices(x) = New StockPrices(r.Next, r.Next.ToString, r.Next.ToString, r.Next.ToString,
                                                       r.Next.ToString, r.Next, r.Next, True,
                                                       New TimeSpan(1, 0, 0), #07/29/2016 2:00 PM#, #07/30/2016 2:00 PM#, r.Next, r.Next,
                                                       r.Next, r.Next, r.Next)
            Next
        End If

        Debug.Print(StoredStockPrices(1).Name.ToString & "|" & StoredStockPrices.Count)
    End Sub

    Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
        Call StartLoops()
    End Sub

    Public Sub StartLoops()
        iLoops = True
        Dim iThreadLoops As New Threading.Thread(New Threading.ThreadStart(AddressOf threadLoops))
        iThreadLoops.Start()
    End Sub

    Private Sub threadLoops()
        If StoredStockPrices.Count > 0 Then
            Do While iLoops = True

                Dim copyStoredStockPrices As New ConcurrentDictionary(Of Integer, StockPrices)
                copyStoredStockPrices = StoredStockPrices

                '360 p/s with 50% CPU usage
                Threading.Tasks.Parallel.ForEach(copyStoredStockPrices.Values.Reverse, Sub(myStockPrices)
                                                                                           Call CheckLowPrices(myStockPrices)
                                                                                       End Sub)



                '450 p/s with 20% CPU usage
                'For Each myStockPrices In copyStoredStockPrices.Values.Reverse
                '    Call CheckLowPrices(myStockPrices)
                'Next

                iTotalLoopsPM += 1
                copyStoredStockPrices = Nothing

                Threading.Thread.Sleep(1)
            Loop
        End If
    End Sub

    Private Sub CheckLowPrices(myStockPrices As StockPrices)
        If myStockPrices.StrikePrice < 10 AndAlso myStockPrices.Index = 10 Then
            If myStockPrices.Sequence < 500 AndAlso myStockPrices.Index < 20 Then
                'submit order
            End If
        End If
    End Sub

    Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
        Dim PerSec As Double = (iTotalLoopsPM / Date.UtcNow.Second)
        Label1.Text = Math.Round(PerSec, 0) & "p/s"
        Label3.Text = Math.Round(iTotalLoopsPM, 0) & "p/m"

        If Date.UtcNow.Second = 0 Then
            iTotalLoopsPM = 0
        End If

        Label2.Text = Date.UtcNow.ToString

        'Stock prices are always changing, hundreds of times per second.
        If StoredStockPrices.Count > 0 Then
            For Each eachRecord In StoredStockPrices
                StoredStockPrices(eachRecord.Key) = New StockPrices(r.Next, r.Next.ToString, r.Next.ToString, r.Next.ToString,
                                                       r.Next.ToString, r.Next, r.Next, True,
                                                       New TimeSpan(1, 0, 0), #07/29/2016 2:00 PM#, #07/30/2016 2:00 PM#, r.Next, r.Next,
                                                       r.Next, r.Next, r.Next)
            Next
        End If
    End Sub
End Class

Open in new window



Public Class StockPrices
    Private m_Index As Integer
    Private m_Symbol As String
    Private m_Name As String
    Private m_ContractStatus As String
    Private m_UnderlyingSymbol As String
    Private m_UnderlyingSettledPrice As Double
    Private m_StrikePrice As Double
    Private m_HasTraded As Boolean
    Private m_LifeSpan As TimeSpan
    Private m_OpensAt As DateTime
    Private m_ClosesAt As DateTime
    Private m_CeilingPrice As Double
    Private m_FloorPrice As Double
    Private m_Latency As Long
    Private m_Sequence As Integer
    Private m_TickMath As Double

    Public Sub New(ByVal iIndex As Integer, ByVal iSymbol As String, ByVal iName As String, ByVal iContractStatus As String,
                   ByVal iUnderlyingSymbol As String, ByVal iUnderlyingSettledPrice As Double, ByVal iStrikePrice As Double,
                   ByVal iHasTraded As Boolean, ByVal iLifeSpan As TimeSpan, ByVal iOpensAt As DateTime, ByVal iClosesAt As DateTime,
                   ByVal iCeilingPrice As Double, ByVal iFloorPrice As Double, ByVal iLatency As Long, ByVal iSequence As Integer,
                   ByVal iTickMath As Double)
        Me.m_Index = iIndex
        Me.m_Symbol = iSymbol
        Me.m_Name = iName
        Me.m_ContractStatus = iContractStatus
        Me.m_UnderlyingSymbol = iUnderlyingSymbol
        Me.m_UnderlyingSettledPrice = iUnderlyingSettledPrice
        Me.m_StrikePrice = iStrikePrice
        Me.m_HasTraded = iHasTraded
        Me.m_LifeSpan = iLifeSpan
        Me.m_OpensAt = iOpensAt
        Me.m_ClosesAt = iClosesAt
        Me.m_CeilingPrice = iCeilingPrice
        Me.m_FloorPrice = iFloorPrice
        Me.m_Latency = iLatency
        Me.m_Sequence = iSequence
        Me.m_TickMath = iTickMath
    End Sub

    Public Property Index() As Integer
        Get
            Return m_Index
        End Get
        Set
            m_Index = Value
        End Set
    End Property

    Public Property Symbol() As String
        Get
            Return m_Symbol
        End Get
        Set
            m_Symbol = Value
        End Set
    End Property

    Public Property Name() As String
        Get
            Return m_Name
        End Get
        Set
            m_Name = Value
        End Set
    End Property

    Public Property ContractStatus() As String
        Get
            Return m_ContractStatus
        End Get
        Set
            m_ContractStatus = Value
        End Set
    End Property

    Public Property UnderlyingSymbol() As String
        Get
            Return m_UnderlyingSymbol
        End Get
        Set
            m_UnderlyingSymbol = Value
        End Set
    End Property

    Public Property UnderlyingSettledPrice() As Double
        Get
            Return m_UnderlyingSettledPrice
        End Get
        Set
            m_UnderlyingSettledPrice = Value
        End Set
    End Property

    Public Property StrikePrice() As Double
        Get
            Return m_StrikePrice
        End Get
        Set
            m_StrikePrice = Value
        End Set
    End Property

    Public Property LifeSpan() As TimeSpan
        Get
            Return m_LifeSpan
        End Get
        Set
            m_LifeSpan = Value
        End Set
    End Property

    Public Property HasTraded() As Boolean
        Get
            Return m_HasTraded
        End Get
        Set
            m_HasTraded = Value
        End Set
    End Property

    Public Property OpensAt() As DateTime
        Get
            Return m_OpensAt
        End Get
        Set
            m_OpensAt = Value
        End Set
    End Property

    Public Property ClosesAt() As DateTime
        Get
            Return m_ClosesAt
        End Get
        Set
            m_ClosesAt = Value
        End Set
    End Property

    Public Property CeilingPrice() As Double
        Get
            Return m_CeilingPrice
        End Get
        Set
            m_CeilingPrice = Value
        End Set
    End Property

    Public Property FloorPrice() As Double
        Get
            Return m_FloorPrice
        End Get
        Set
            m_FloorPrice = Value
        End Set
    End Property

    Public Property Latency() As Long
        Get
            Return m_Latency
        End Get
        Set
            m_Latency = Value
        End Set
    End Property

    Public Property Sequence() As Integer
        Get
            Return m_Sequence
        End Get
        Set
            m_Sequence = Value
        End Set
    End Property

    Public Property TickMath() As Double
        Get
            Return m_TickMath
        End Get
        Set
            m_TickMath = Value
        End Set
    End Property
End Class

Open in new window

0
Comment
Question by:XK8ER
5 Comments
 
LVL 78

Expert Comment

by:David Johnson, CD, MVP
ID: 41736125
have you run your code through a profiler and found any bottlenecks?
0
 
LVL 1

Author Comment

by:XK8ER
ID: 41736195
yes I have used profiler but the issue is not bottlenecks is speed
0
 
LVL 142

Accepted Solution

by:
Guy Hengel [angelIII / a3] earned 250 total points
ID: 41736448
the only effective way is to have this data the "calculated"(means ready) as the collection is filled/updated, and not afterwards.
this may slower a bit the insert/update of items in the collection, but in the end be the fastest solution.
you may try to either override the class to implement the necessary functions/data, or if that is not possible, creat a class that incorporates the class ...
you will need to override/enhance the item adding and updating process accordingly.
just trying to get the looping will not give any good results.
compare it like if the data is a paper folder, without an index(es) . trying to find some item will require you to read through the entire folder.
with the indexes in place, you only consult the index
adding/updating/removing items needs to update the index

hope this helps
0
 
LVL 32

Assisted Solution

by:sarabande
sarabande earned 250 total points
ID: 41736733
If myStockPrices.StrikePrice < 10 AndAlso myStockPrices.Index = 10 Then
            If myStockPrices.Sequence < 500 AndAlso myStockPrices.Index < 20 Then
                'submit order
            End If
        End If

actually i don't understand this statement: if myStockPrices.Index = 10 then obviously myStockPrices.Index < 20 is also true.

so in my opinion it makes less sense to have a permanent loop of 15,000 cycles which checks the same conditions again and again. instead you should issue an action when any of the inputs myStockPrices.Index, myStockPrices.StrikePrice, or myStockPrices.Sequence was modified.

Sara
0
 
LVL 1

Author Comment

by:XK8ER
ID: 41736782
there are two connections to two different market data feed.
the function CheckLowPrices is just something very similar to what we have in place.

example:
data feed #1 says EUR/USD contract 1.1166 is selling for $50
data feed #2 says the underlying price for EUR/USD is at 1.11652

if we only check one time it could work yes but sometimes data feed #1 doesn't move for several second so if its a good price according to data feed #2 then we miss out the window to submit a new order, and when market data #1 moves again its already not a good price
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now