Solved

speed up additem

Posted on 2002-05-10
35
600 Views
Last Modified: 2010-05-02
I have over 66000 text line (reading from ASCII file) wants to display on the list box and each line is variable length.

The following is my routine.

Public Sub AddToListBox()
    Dim Mess() As String, I As Long, Counter As Long
    Dim Tmp As String, Length As Long, ListHwnd As Long, TextSize As Long

    On Error Resume Next
    LstMessages.Clear
    LstMessages.SetFocus
    ListHwnd = GetFocus()
    LstMessages.Visible = False
    DoEvents
    Me.ScaleMode = 3
    Mess = Split(Messages, vbCrLf)      ' contains all lines with vbcrlf delimiter
    Counter = UBound(Mess)
    Call LockWindowUpdate(ListHwnd)
    For I = 1 To Counter + 1
        Tmp = Mess(I - 1)
        SendMessage ListHwnd, LB_ADDSTRING, 0, ByVal Tmp
        Length = Me.TextWidth(Tmp) + 20
        TextSize = IIf(Length > TextSize, Length, TextSize)
    Next
    Call LockWindowUpdate(0&)
    SendMessage ListHwnd, LB_SETHORIZONTALEXTENT, TextSize, 0&
    LstMessages.Visible = True
    Me.ScaleMode = 1
End Sub


Even I replace the line
SendMessage ListHwnd, LB_ADDSTRING, 0, ByVal Tmp
with
LstMessages.additem Tmp

It tooks over 2 min on my computer (800MHz PIII).
Is any way that it can speed up the additem or using sendmessage.

I would not like to use any third party listbox.




Any idea?
0
Comment
Question by:EDDYKT
  • 12
  • 6
  • 5
  • +8
35 Comments
 
LVL 69

Expert Comment

by:Éric Moreau
Comment Utility
are you sure that you need 66000 lines in a listbox? what is the purpose of this listbox?
0
 
LVL 75

Expert Comment

by:Anthony Perkins
Comment Utility
IMHO you have about 65,800 items too many.  It is time to re-design.

Anthony
0
 
LVL 28

Expert Comment

by:AzraSound
Comment Utility
Gotta agree with everyone here...design needs reworking, not the functionality or code.  Can these items be further separated into groups?  Can they be fed to a database for quicker retrieval later?
0
 
LVL 16

Expert Comment

by:Richie_Simonetti
Comment Utility
I must to agree too.
At this point, API is the fastest you can reach.
0
 
LVL 12

Expert Comment

by:roverm
Comment Utility
I agree as well: Load the first (let's say) 200, then use a background thread to continue the loading of the rest.

D'Mzzl!
RoverM
0
 
LVL 22

Expert Comment

by:rspahitz
Comment Utility
Rather than jump on the bandwagon...oh, what the heck...image the poor use who has to scroll through 66,000 items to find the desired one.

How about loading everything into a multi-line textbox and whenever the user clicks, determine the current location to determine the "selected" line?

Regardless of your implementation, how can you expect 66,000 lines of variable-length text to be split into small pieces and gathered together quickly?  String parsing is usually the slowest thing a computer does.
0
 
LVL 15

Expert Comment

by:ameba
Comment Utility
My listbox can load only 32768 items.
If speed is important, you don't want to use Me.TextWidth or Iif.
GetFocus() - this was needed in VB1, now listboxes have hWnd property
If you use LockWindowUpdate, you don't need to set .Visible property
0
 
LVL 16

Expert Comment

by:Richie_Simonetti
Comment Utility
As ameba pointed, not use textwidth. If you need to call
SendMessage ListHwnd, LB_SETHORIZONTALEXTENT, TextSize, 0&
use the loop on a similar fashion:

Dim ret As Long, sItmText As String
Dim i As Long, lResult As Long, lItmMax As Long
With List1
    For i = 0 To .ListCount - 1
        lResult = SendMessage(.hwnd, LB_GETTEXTLEN, i, ByVal 0)
        If (lResult > lItmMax) Then
            sItmText = .List(i)
            lItmMax = lResult
        End If
    Next i
    ret = SendMessage(.hwnd, LB_SETHORIZONTALEXTENT, lResult, ByVal 0)
End With
even adding it to your existing loop.
But take in mind all other comments regardind redesing.
0
 
LVL 2

Expert Comment

by:selim007
Comment Utility
1st if ur source is a text file try using the Memory Mapping Method which will accelerate the reading process from the file.
2nd there's no way to load these items in a listbox so fast.
beside the list box won't be able to support that number of items unless you wanna design ur own one
y don't u try integrating HTML under your VB form.
i mean simply use the Microsoft Internet Controls OCX in your application, the load your file as HTML and make a link for each item. of course it's ur code that must create the link and obviously u must use some scripts like java or vb script... it 's not an easy way but it works if u r insisting on having 66000 items

0
 
LVL 1

Expert Comment

by:olx
Comment Utility
EDDYKT

please hide the listbox before add items to it.
show it again when the adding ends.

doing that it will take from 39 to 8 seconds to add 30000 things to a listbox

but i still think in rspahitz comment "image the poor use who has to scroll through 66,000 items to find the desired one."

hope this helps
0
 
LVL 1

Expert Comment

by:olx
Comment Utility
i personally think that additem is faster.
but ask Azrasound about it.
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
OK, Let me explain more,
I've an OPC application that print out the system messages (error message) into the output log file. When the new log file exceed 1M size, I will delete the old one and rename the new to old log file.

I've another VB program that read both old and new ASCII file and display on the listbox. Each message will be timestamp and the user should be easy to go through the list box to check the progress of the program.

To AzraSound:
I did try to use database but the write performance from OPC application is poor. However the read is fast if I use database grid. The OPC application should be fast enough to write all log to log file without any delay.

To roverm:
I'm not sure what do you meant here. If I can use timer, the text in listbox must flicking when I add the new text.

To ameba, Richie_Simonetti:
If I have 66000 lines, it will take a lot amoiunt of time to go through each textline and find out the longest line width.
I can take out thr textwidth and try.

To olx:
I did set the listbox to invisible.
0
 
LVL 16

Expert Comment

by:Richie_Simonetti
Comment Utility
But you could get the actual length of every line when you are populating the list in first place!
0
 
LVL 15

Expert Comment

by:ameba
Comment Utility
Who needs precise width of the line - make it some big fixed value, or use: len() * textwidth("W")

Did you know that "On Error Resume Next" hides all errors?
Are you sure your listbox can hold more than 32768 items?
0
 
LVL 16

Expert Comment

by:Richie_Simonetti
Comment Utility
Well, i have to put some ugly stuff to try the code :


Option Explicit
Private Type SIZE
    cx As Long
    cy As Long
End Type

Private Const LB_ADDSTRING = &H180
Private Const LB_SETHORIZONTALEXTENT = &H194
Private Const LB_GETTEXTLEN = &H18A
Private Const WM_GETFONT = &H31
Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" (ByVal hwnd As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long
Private Declare Function ReleaseDC Lib "user32" (ByVal hwnd As Long, ByVal hdc As Long) As Long

Private Declare Function SelectObject Lib "gdi32" (ByVal hdc As Long, ByVal hObject As Long) As Long
Private Declare Function GetDC Lib "user32" (ByVal hwnd As Long) As Long
Private Declare Function GetTextExtentPoint32 Lib "gdi32" Alias "GetTextExtentPoint32A" (ByVal hdc As Long, ByVal lpsz As String, ByVal cbString As Long, lpSize As SIZE) As Long

Public Sub AddToListBox(messages As String)
Dim Mess() As String, I As Long, Counter As Long
Dim lItm As Long
Dim lResult As Long, lItemmax As Long, sString As String
Dim uSize As SIZE
Dim lstDC As Long, lFont As Long, lFontOld As Long
On Error Resume Next
With lstMessages

     .Clear
     .Visible = False
     Mess = Split(messages, vbCrLf)      ' contains all lines with vbcrlf delimiter
     Counter = UBound(Mess)
     For I = 0 To Counter - 1
        lItm = SendMessage(.hwnd, LB_ADDSTRING, 0, ByVal Mess(I))
        lResult = SendMessage(.hwnd, LB_GETTEXTLEN, _
                          lItm, ByVal 0)
        If (lResult > lItemmax) Then
            sString = Mess(I)
            lItemmax = lResult
        End If
    Next
   
    lstDC = GetDC(.hwnd)
    lFont = SendMessage(.hwnd, WM_GETFONT, 0, ByVal 0)
    'Select the font in to the device context, and retain prior font
    lFontOld = SelectObject(hdc, lFont)
    lResult = GetTextExtentPoint32(hdc, sString, lResult, uSize)
    SelectObject hdc, lFontOld
    ReleaseDC .hwnd, hdc
    SendMessage .hwnd, LB_SETHORIZONTALEXTENT, uSize.cx, 0&
    .Visible = True
End With
   
End Sub

Private Sub Form_Load()
AddToListBox "n,mdnf,nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,ttftftftfdsn,ds" & vbCrLf & _
            "n,mdnf,nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,ncfdedsds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,1234543nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,cdxdsnds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,nds,fnsfn,dsfn,dsn,ds" & vbCrLf & _
            "n,mdnf,nds,knmmmmmmmmmmmmjhygtffnsfn,dsfn,dsn,ds" & vbCrLf
End Sub


0
 
LVL 28

Expert Comment

by:AzraSound
Comment Utility
>>I did try to use database but the write performance from OPC application is poor.

Did you ever consider writing to the DB instead of a log file?  An insert into the DB is quick, you would have more ability to query the log, maintain a longer history of messages logged by the OPC server, allow for simple searches, etc.
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
OK, Let me explain more,
I've an OPC application that print out the system messages (error message) into the output log file. When the new log file exceed 1M size, I will delete the old one and rename the new to old log file.

I've another VB program that read both old and new ASCII file and display on the listbox. Each message will be timestamp and the user should be easy to go through the list box to check the progress of the program.

To AzraSound:
I did try to use database but the write performance from OPC application is poor. However the read is fast if I use database grid. The OPC application should be fast enough to write all log to log file without any delay.

To roverm:
I'm not sure what do you meant here. If I can use timer, the text in listbox must flicking when I add the new text.

To ameba, Richie_Simonetti:
If I have 66000 lines, it will take a lot amoiunt of time to go through each textline and find out the longest line width.
I can take out thr textwidth and try.

To olx:
I did set the listbox to invisible.
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
Don't know what happen to last message?->

To ameba:
>>Who needs precise width of the line - make it some big fixed value, or use: len() * textwidth

I will get back to this and try out.

Yes it work over the magic number. I am really sure because I put the line number in front of each line.

To AzraSound:
It treat each log message is separated so that each update message will require open database, open connection, write data and close database. What do you think the speed is.

To Richie_Simonetti:
I'wll try this and let you know.

To ameba:
>>Who needs precise width of the line - make it some big fixed value, or use: len() * textwidth

I will get back to this and try out
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
By the way, I use windows 2000, vb6 sp5
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
I try all the method and event comment out the textwidth line. The result is almost the same; just 1 sec faster.

I believe it is not textwidth to slow it down. It is additem.

Any suggestion

Should I try using datasource property?
0
 
LVL 28

Expert Comment

by:AzraSound
Comment Utility
>>It treat each log message is separated so that each update message will require open database, open connection, write data and close database. What do you think the speed is.

It won't be so slow that you will notice a huge performance loss.  Then, you have the added bonus of quickly searching this database based on the user's preferences.  For example, instead of just coming up and showing every entry in the log, they can enter in maybe a start and end date, messages that contain a certain message, critical messages only, etc.  Plus, then you have an easy means of allowing them to delete all log entries, delete selected entries, etc (all the benefits of working with data in a database).
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
AzraSound,

I think of that already. The main reason we drop the database because we may need to write 20 or more messsages in a second into the log file (if required). Using database is not a proper method to do. Do you think so?

The other problem is we have to maintain how many line in the database so that I have to delete the old one before I added the new one in. To doing so the database file size will keep growing as well and I have to compact the database once a while. I don't think we have time to do that in OPC server.

Those all kinds of problems will lead us to drop database method.
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
rspahitz :

I tried using multi-text box. I add set my whole text string (text wiht vbcrlf delimiter) into the textbox in one short but the alignment doesn't look really good on text box

8-<
0
 
LVL 28

Expert Comment

by:AzraSound
Comment Utility
Database speed may be quicker than you think, and if you are hitting the database that often, you could probably just leave a connection open always (and remove the performance issue of opening/closing that connection on every call).  I imagine you would use a full rich database engine like SQL Server.


>>The other problem is we have to maintain how many line in the database so that I have to delete the old one before I added the new one in.

Thats where you could use the rich features of SQL Server.  You can schedule it to run a script every so often (determined by you) that would go and remove records as needed.  This would not require your OPC server to do anything regarding that since it would be running in a separate process (SQL Server's) and executed on a schedule.
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
AzraSound,

For those small job, I should mention that I can only use MS access database. SQL server may be a little bit over kill. 8->
0
 
LVL 28

Expert Comment

by:AzraSound
Comment Utility
Oh, I see...
Are you writing an OPC client or OPC server?
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
OPC Server
0
 
LVL 15

Accepted Solution

by:
ameba earned 200 total points
Comment Utility
It takes 6-7 seconds on Celeron /600 MHz.
It takes longer if I have many applications running (because of swapping to disk).
Here is non-optimized code which adds 66000 items to listview:

' Form1, add listview and two buttons
Option Explicit
Private Declare Function GetTickCount Lib "kernel32" () As Long

Private Sub Form_Load()
    Command1.Caption = "create file"
    Command2.Caption = "read"
    With Me.ListView1
        .ColumnHeaders.Add , , "Message", 30000
        .View = lvwReport
    End With
End Sub

' create test file
Private Sub Command1_Click()
    Dim i As Long, ff As Integer, filename As String
    filename = "c:\test.txt"
   
    ff = FreeFile
    Open filename For Output As #ff
   
    For i = 1 To 66000
        Print #ff, i & " " & String(150, "a")
    Next
   
    Close #ff
   
    Caption = Format(filelen(filename), "#,##0")
End Sub

' read file and fill listview
Private Sub Command2_Click()
    Dim i As Long, ff As Integer, filename As String, filelen As Long
    Dim s As String, msg() As String, tim0 As Long
   
    tim0 = GetTickCount ' start timer
   
    ' read file into big string
    filename = "c:\test.txt"
    ff = FreeFile
    Open filename For Binary Access Read Lock Read Write As #ff
    s = Space$(LOF(ff))
    Get #ff, , s
    Close ff
   
    ' get array of lines
    msg = Split(s, vbCrLf)
    s = ""
   
    ' fill list
    Me.ListView1.Visible = False
    With Me.ListView1.ListItems
        .Clear
        For i = 0 To UBound(msg) - 1
            .Add , , Left(msg(i), 257)
        Next
    End With
    Me.ListView1.Visible = True
   
    Caption = (GetTickCount - tim0) / 1000 & " seconds"
End Sub
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
Wow, Listview is lot lot fastest than listbox.
That's great. Thanks ameba

Do you know what is the reason?

0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
Good thinking
0
 
LVL 15

Expert Comment

by:ameba
Comment Utility
Thanks.
>Do you know what is the reason?
Maybe Sorted property of the listbox was set to True?
0
 
LVL 15

Expert Comment

by:ameba
Comment Utility
' here is how to set width of the column to tmpwidth (PIXELS)
Private Const LVM_SETCOLUMNWIDTH = (LVM_FIRST + 30)
ret = SendMessage(LV.hwnd, LVM_SETCOLUMNWIDTH, 0, ByVal tmpwidth)

' there is also another msg to set column width automatically...
0
 
LVL 15

Expert Comment

by:ameba
Comment Utility
>set column width automatically...

Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" (ByVal hwnd _
   As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long
Private Const LVM_SETCOLUMNWIDTH = &H1000 + 30
Private Const LVSCW_AUTOSIZE = -1
Private Const LVSCW_AUTOSIZE_USEHEADER = -2

Private Sub Form_Click()
   AutoSizeColumns Me.Listview1
End Sub

Private Sub AutoSizeColumns(Listview As Listview, Optional ByVal UseHeader As Boolean = False)
   Dim i As Integer, lParam As Long
   If UseHeader = False Then
       lParam = LVSCW_AUTOSIZE
   Else
       lParam = LVSCW_AUTOSIZE_USEHEADER
   End If
   For i = 0 To Listview.ColumnHeaders.Count - 1
       SendMessage Listview.hwnd, LVM_SETCOLUMNWIDTH, i, ByVal lParam
   Next
End Sub
0
 
LVL 26

Author Comment

by:EDDYKT
Comment Utility
>>Maybe Sorted property of the listbox was set to True?

No , it is false  8->

Anyway thanks
0
 
LVL 5

Expert Comment

by:Kimpan
Comment Utility
if the control is not visible during the operations, it will go much smoother.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

There are many ways to remove duplicate entries in an SQL or Access database. Most make you temporarily insert an ID field, make a temp table and copy data back and forth, and/or are slow. Here is an easy way in VB6 using ADO to remove duplicate row…
Background What I'm presenting in this article is the result of 2 conditions in my work area: We have a SQL Server production environment but no development or test environment; andWe have an MS Access front end using tables in SQL Server but we a…
Get people started with the process of using Access VBA to control Outlook using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Microsoft Outlook. Using automation, an Access applic…
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now