• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 648
  • Last Modified:

speed up additem

I have over 66000 text line (reading from ASCII file) wants to display on the list box and each line is variable length.

The following is my routine.

Public Sub AddToListBox()
    Dim Mess() As String, I As Long, Counter As Long
    Dim Tmp As String, Length As Long, ListHwnd As Long, TextSize As Long

    On Error Resume Next
    LstMessages.Clear
    LstMessages.SetFocus
    ListHwnd = GetFocus()
    LstMessages.Visible = False
    DoEvents
    Me.ScaleMode = 3
    Mess = Split(Messages, vbCrLf)      ' contains all lines with vbcrlf delimiter
    Counter = UBound(Mess)
    Call LockWindowUpdate(ListHwnd)
    For I = 1 To Counter + 1
        Tmp = Mess(I - 1)
        SendMessage ListHwnd, LB_ADDSTRING, 0, ByVal Tmp
        Length = Me.TextWidth(Tmp) + 20
        TextSize = IIf(Length > TextSize, Length, TextSize)
    Next
    Call LockWindowUpdate(0&)
    SendMessage ListHwnd, LB_SETHORIZONTALEXTENT, TextSize, 0&
    LstMessages.Visible = True
    Me.ScaleMode = 1
End Sub


Even I replace the line
SendMessage ListHwnd, LB_ADDSTRING, 0, ByVal Tmp
with
LstMessages.additem Tmp

It tooks over 2 min on my computer (800MHz PIII).
Is any way that it can speed up the additem or using sendmessage.

I would not like to use any third party listbox.




Any idea?
0
EDDYKT
Asked:
EDDYKT
  • 12
  • 6
  • 5
  • +8
1 Solution
 
Éric MoreauSenior .Net ConsultantCommented:
are you sure that you need 66000 lines in a listbox? what is the purpose of this listbox?
0
 
Anthony PerkinsCommented:
IMHO you have about 65,800 items too many.  It is time to re-design.

Anthony
0
 
AzraSoundCommented:
Gotta agree with everyone here...design needs reworking, not the functionality or code.  Can these items be further separated into groups?  Can they be fed to a database for quicker retrieval later?
0
Cloud Class® Course: MCSA MCSE Windows Server 2012

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

 
Richie_SimonettiIT OperationsCommented:
I must to agree too.
At this point, API is the fastest you can reach.
0
 
rovermCommented:
I agree as well: Load the first (let's say) 200, then use a background thread to continue the loading of the rest.

D'Mzzl!
RoverM
0
 
rspahitzCommented:
Rather than jump on the bandwagon...oh, what the heck...image the poor use who has to scroll through 66,000 items to find the desired one.

How about loading everything into a multi-line textbox and whenever the user clicks, determine the current location to determine the "selected" line?

Regardless of your implementation, how can you expect 66,000 lines of variable-length text to be split into small pieces and gathered together quickly?  String parsing is usually the slowest thing a computer does.
0
 
amebaCommented:
My listbox can load only 32768 items.
If speed is important, you don't want to use Me.TextWidth or Iif.
GetFocus() - this was needed in VB1, now listboxes have hWnd property
If you use LockWindowUpdate, you don't need to set .Visible property
0
 
Richie_SimonettiIT OperationsCommented:
As ameba pointed, not use textwidth. If you need to call
SendMessage ListHwnd, LB_SETHORIZONTALEXTENT, TextSize, 0&
use the loop on a similar fashion:

Dim ret As Long, sItmText As String
Dim i As Long, lResult As Long, lItmMax As Long
With List1
    For i = 0 To .ListCount - 1
        lResult = SendMessage(.hwnd, LB_GETTEXTLEN, i, ByVal 0)
        If (lResult > lItmMax) Then
            sItmText = .List(i)
            lItmMax = lResult
        End If
    Next i
    ret = SendMessage(.hwnd, LB_SETHORIZONTALEXTENT, lResult, ByVal 0)
End With
even adding it to your existing loop.
But take in mind all other comments regardind redesing.
0
 
selim007Commented:
1st if ur source is a text file try using the Memory Mapping Method which will accelerate the reading process from the file.
2nd there's no way to load these items in a listbox so fast.
beside the list box won't be able to support that number of items unless you wanna design ur own one
y don't u try integrating HTML under your VB form.
i mean simply use the Microsoft Internet Controls OCX in your application, the load your file as HTML and make a link for each item. of course it's ur code that must create the link and obviously u must use some scripts like java or vb script... it 's not an easy way but it works if u r insisting on having 66000 items

0
 
olxCommented:
EDDYKT

please hide the listbox before add items to it.
show it again when the adding ends.

doing that it will take from 39 to 8 seconds to add 30000 things to a listbox

but i still think in rspahitz comment "image the poor use who has to scroll through 66,000 items to find the desired one."

hope this helps
0
 
olxCommented:
i personally think that additem is faster.
but ask Azrasound about it.
0
 
EDDYKTAuthor Commented:
OK, Let me explain more,
I've an OPC application that print out the system messages (error message) into the output log file. When the new log file exceed 1M size, I will delete the old one and rename the new to old log file.

I've another VB program that read both old and new ASCII file and display on the listbox. Each message will be timestamp and the user should be easy to go through the list box to check the progress of the program.

To AzraSound:
I did try to use database but the write performance from OPC application is poor. However the read is fast if I use database grid. The OPC application should be fast enough to write all log to log file without any delay.

To roverm:
I'm not sure what do you meant here. If I can use timer, the text in listbox must flicking when I add the new text.

To ameba, Richie_Simonetti:
If I have 66000 lines, it will take a lot amoiunt of time to go through each textline and find out the longest line width.
I can take out thr textwidth and try.

To olx:
I did set the listbox to invisible.
0
 
Richie_SimonettiIT OperationsCommented:
But you could get the actual length of every line when you are populating the list in first place!
0
 
amebaCommented:
Who needs precise width of the line - make it some big fixed value, or use: len() * textwidth("W")

Did you know that "On Error Resume Next" hides all errors?
Are you sure your listbox can hold more than 32768 items?
0
 
Richie_SimonettiIT OperationsCommented:
Well, i have to put some ugly stuff to try the code :


Option Explicit
Private Type SIZE
    cx As Long
    cy As Long
End Type

Private Const LB_ADDSTRING = &H180
Private Const LB_SETHORIZONTALEXTENT = &H194
Private Const LB_GETTEXTLEN = &H18A
Private Const WM_GETFONT = &H31
Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" (ByVal hwnd As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long
Private Declare Function ReleaseDC Lib "user32" (ByVal hwnd As Long, ByVal hdc As Long) As Long

Private Declare Function SelectObject Lib "gdi32" (ByVal hdc As Long, ByVal hObject As Long) As Long
Private Declare Function GetDC Lib "user32" (ByVal hwnd As Long) As Long
Private Declare Function GetTextExtentPoint32 Lib "gdi32" Alias "GetTextExtentPoint32A" (ByVal hdc As Long, ByVal lpsz As String, ByVal cbString As Long, lpSize As SIZE) As Long

Public Sub AddToListBox(messages As String)
Dim Mess() As String, I As Long, Counter As Long
Dim lItm As Long
Dim lResult As Long, lItemmax As Long, sString As String
Dim uSize As SIZE
Dim lstDC As Long, lFont As Long, lFontOld As Long
On Error Resume Next
With lstMessages

     .Clear
     .Visible = False
     Mess = Split(messages, vbCrLf)      ' contains all lines with vbcrlf delimiter
     Counter = UBound(Mess)
     For I = 0 To Counter - 1
        lItm = SendMessage(.hwnd, LB_ADDSTRING, 0, ByVal Mess(I))
        lResult = SendMessage(.hwnd, LB_GETTEXTLEN, _
                          lItm, ByVal 0)
        If (lResult > lItemmax) Then
            sString = Mess(I)
            lItemmax = lResult
        End If
    Next
   
    lstDC = GetDC(.hwnd)
    lFont = SendMessage(.hwnd, WM_GETFONT, 0, ByVal 0)
    'Select the font in to the device context, and retain prior font
    lFontOld = SelectObject(hdc, lFont)
    lResult = GetTextExtentPoint32(hdc, sString, lResult, uSize)
    SelectObject hdc, lFontOld
    ReleaseDC .hwnd, hdc
    SendMessage .hwnd, LB_SETHORIZONTALEXTENT, uSize.cx, 0&
    .Visible = True
End With
   
End Sub

Private Sub Form_Load()
AddToListBox "n,mdnf,nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,ttftftftfdsn,ds" & vbCrLf & _
            "n,mdnf,nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,ncfdedsds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,1234543nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,cdxdsnds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,nds,knmmmmmmmmmmmmjhygtffnsfn,dsfn,dsn,ds" & vbCrLf
End Sub


0
 
AzraSoundCommented:
>>I did try to use database but the write performance from OPC application is poor.

Did you ever consider writing to the DB instead of a log file?  An insert into the DB is quick, you would have more ability to query the log, maintain a longer history of messages logged by the OPC server, allow for simple searches, etc.
0
 
EDDYKTAuthor Commented:
OK, Let me explain more,
I've an OPC application that print out the system messages (error message) into the output log file. When the new log file exceed 1M size, I will delete the old one and rename the new to old log file.

I've another VB program that read both old and new ASCII file and display on the listbox. Each message will be timestamp and the user should be easy to go through the list box to check the progress of the program.

To AzraSound:
I did try to use database but the write performance from OPC application is poor. However the read is fast if I use database grid. The OPC application should be fast enough to write all log to log file without any delay.

To roverm:
I'm not sure what do you meant here. If I can use timer, the text in listbox must flicking when I add the new text.

To ameba, Richie_Simonetti:
If I have 66000 lines, it will take a lot amoiunt of time to go through each textline and find out the longest line width.
I can take out thr textwidth and try.

To olx:
I did set the listbox to invisible.
0
 
EDDYKTAuthor Commented:
Don't know what happen to last message?->

To ameba:
>>Who needs precise width of the line - make it some big fixed value, or use: len() * textwidth

I will get back to this and try out.

Yes it work over the magic number. I am really sure because I put the line number in front of each line.

To AzraSound:
It treat each log message is separated so that each update message will require open database, open connection, write data and close database. What do you think the speed is.

To Richie_Simonetti:
I'wll try this and let you know.

To ameba:
>>Who needs precise width of the line - make it some big fixed value, or use: len() * textwidth

I will get back to this and try out
0
 
EDDYKTAuthor Commented:
By the way, I use windows 2000, vb6 sp5
0
 
EDDYKTAuthor Commented:
I try all the method and event comment out the textwidth line. The result is almost the same; just 1 sec faster.

I believe it is not textwidth to slow it down. It is additem.

Any suggestion

Should I try using datasource property?
0
 
AzraSoundCommented:
>>It treat each log message is separated so that each update message will require open database, open connection, write data and close database. What do you think the speed is.

It won't be so slow that you will notice a huge performance loss.  Then, you have the added bonus of quickly searching this database based on the user's preferences.  For example, instead of just coming up and showing every entry in the log, they can enter in maybe a start and end date, messages that contain a certain message, critical messages only, etc.  Plus, then you have an easy means of allowing them to delete all log entries, delete selected entries, etc (all the benefits of working with data in a database).
0
 
EDDYKTAuthor Commented:
AzraSound,

I think of that already. The main reason we drop the database because we may need to write 20 or more messsages in a second into the log file (if required). Using database is not a proper method to do. Do you think so?

The other problem is we have to maintain how many line in the database so that I have to delete the old one before I added the new one in. To doing so the database file size will keep growing as well and I have to compact the database once a while. I don't think we have time to do that in OPC server.

Those all kinds of problems will lead us to drop database method.
0
 
EDDYKTAuthor Commented:
rspahitz :

I tried using multi-text box. I add set my whole text string (text wiht vbcrlf delimiter) into the textbox in one short but the alignment doesn't look really good on text box

8-<
0
 
AzraSoundCommented:
Database speed may be quicker than you think, and if you are hitting the database that often, you could probably just leave a connection open always (and remove the performance issue of opening/closing that connection on every call).  I imagine you would use a full rich database engine like SQL Server.


>>The other problem is we have to maintain how many line in the database so that I have to delete the old one before I added the new one in.

Thats where you could use the rich features of SQL Server.  You can schedule it to run a script every so often (determined by you) that would go and remove records as needed.  This would not require your OPC server to do anything regarding that since it would be running in a separate process (SQL Server's) and executed on a schedule.
0
 
EDDYKTAuthor Commented:
AzraSound,

For those small job, I should mention that I can only use MS access database. SQL server may be a little bit over kill. 8->
0
 
AzraSoundCommented:
Oh, I see...
Are you writing an OPC client or OPC server?
0
 
EDDYKTAuthor Commented:
OPC Server
0
 
amebaCommented:
It takes 6-7 seconds on Celeron /600 MHz.
It takes longer if I have many applications running (because of swapping to disk).
Here is non-optimized code which adds 66000 items to listview:

' Form1, add listview and two buttons
Option Explicit
Private Declare Function GetTickCount Lib "kernel32" () As Long

Private Sub Form_Load()
    Command1.Caption = "create file"
    Command2.Caption = "read"
    With Me.ListView1
        .ColumnHeaders.Add , , "Message", 30000
        .View = lvwReport
    End With
End Sub

' create test file
Private Sub Command1_Click()
    Dim i As Long, ff As Integer, filename As String
    filename = "c:\test.txt"
   
    ff = FreeFile
    Open filename For Output As #ff
   
    For i = 1 To 66000
        Print #ff, i & " " & String(150, "a")
    Next
   
    Close #ff
   
    Caption = Format(filelen(filename), "#,##0")
End Sub

' read file and fill listview
Private Sub Command2_Click()
    Dim i As Long, ff As Integer, filename As String, filelen As Long
    Dim s As String, msg() As String, tim0 As Long
   
    tim0 = GetTickCount ' start timer
   
    ' read file into big string
    filename = "c:\test.txt"
    ff = FreeFile
    Open filename For Binary Access Read Lock Read Write As #ff
    s = Space$(LOF(ff))
    Get #ff, , s
    Close ff
   
    ' get array of lines
    msg = Split(s, vbCrLf)
    s = ""
   
    ' fill list
    Me.ListView1.Visible = False
    With Me.ListView1.ListItems
        .Clear
        For i = 0 To UBound(msg) - 1
            .Add , , Left(msg(i), 257)
        Next
    End With
    Me.ListView1.Visible = True
   
    Caption = (GetTickCount - tim0) / 1000 & " seconds"
End Sub
0
 
EDDYKTAuthor Commented:
Wow, Listview is lot lot fastest than listbox.
That's great. Thanks ameba

Do you know what is the reason?

0
 
EDDYKTAuthor Commented:
Good thinking
0
 
amebaCommented:
Thanks.
>Do you know what is the reason?
Maybe Sorted property of the listbox was set to True?
0
 
amebaCommented:
' here is how to set width of the column to tmpwidth (PIXELS)
Private Const LVM_SETCOLUMNWIDTH = (LVM_FIRST + 30)
ret = SendMessage(LV.hwnd, LVM_SETCOLUMNWIDTH, 0, ByVal tmpwidth)

' there is also another msg to set column width automatically...
0
 
amebaCommented:
>set column width automatically...

Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" (ByVal hwnd _
   As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long
Private Const LVM_SETCOLUMNWIDTH = &H1000 + 30
Private Const LVSCW_AUTOSIZE = -1
Private Const LVSCW_AUTOSIZE_USEHEADER = -2

Private Sub Form_Click()
   AutoSizeColumns Me.Listview1
End Sub

Private Sub AutoSizeColumns(Listview As Listview, Optional ByVal UseHeader As Boolean = False)
   Dim i As Integer, lParam As Long
   If UseHeader = False Then
       lParam = LVSCW_AUTOSIZE
   Else
       lParam = LVSCW_AUTOSIZE_USEHEADER
   End If
   For i = 0 To Listview.ColumnHeaders.Count - 1
       SendMessage Listview.hwnd, LVM_SETCOLUMNWIDTH, i, ByVal lParam
   Next
End Sub
0
 
EDDYKTAuthor Commented:
>>Maybe Sorted property of the listbox was set to True?

No , it is false  8->

Anyway thanks
0
 
KimpanCommented:
if the control is not visible during the operations, it will go much smoother.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

  • 12
  • 6
  • 5
  • +8
Tackle projects and never again get stuck behind a technical roadblock.
Join Now