Listview - Delay when Unloading

try doing it manually in the unload:

Private Declare Function LockWindowUpdate Lib "user32" Alias "LockWindowUpdate" (ByVal hwndLock As Long) As Long

Dim i As Integer
With Form1.ListView.ListItems
    Call LockWindowUpdate(Form1.ListView.hwnd)
    For i = .Count to 1 Step -1
        Call .Remove(i)
        If (i mod 100 = 0) Then DoEvents
    Next
    Call LockWindowUpdate(0)
End With

i had the same problem with the treeview and this code proved useful

Yes, I know faster way(s).

Hi

Private Declare Function SendMessage Lib "User32" Alias "SendMessageA" (ByVal hwnd As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long
Private Declare Function LockWindowUpdate Lib "User32" (ByVal hwndLock As Long) As Long
Const LVM_FIRST As Long = &H1000
Const LVM_DELETEALLITEMS As Long = (LVM_FIRST + 9)
Private Sub Form_Load()
  With ListView1
  .View = lvwReport
    .ColumnHeaders.Add , , "Item Column"
    .ColumnHeaders.Add , , "Subitem 1"
    .ColumnHeaders.Add , , "Subitem 2"
    Dim i&
    For i = 1 To 3000
      With .ListItems.Add(, , "Item " & CStr(i))
        .SubItems(1) = "Subitem 1"
        .SubItems(2) = "Subitem 2"
      End With
    Next
  End With
End Sub

Private Sub Form_Unload(Cancel As Integer)
   Call LockWindowUpdate(ListView1.hwnd)
   Call SendMessage(ListView1.hwnd, LVM_DELETEALLITEMS, 0&, ByVal 0&)
   Call LockWindowUpdate(0&)
End Sub

Cheers

I tried the suggestions of both AzraSound and Ark. Unfortunately they offered no improvement. AzraSounds approach was very much slower than just doing a .clear. Arks approach was about the same, or slightly slower than a .clear. I think maybe it is to do with the large number of entries in the Listview (50,000+)

Ameba, what is your suggestion?

Adjusted points from 200 to 500

>large number of entries in the Listview (50,000+)

Are you sure it is 50000, my ListView can hold only 32767 listitems (last time I checked)?

Ark's suggestion (DELETEALLITEMS message) is ~2.5 times faster than .Clear

I use the same message, but I have some additional code if ListView is *sorted*. Are you sorting your columns?

Yes I am certain. List and Combobox are restricted to 32767, but the Listview isn't.

Yes the Listview is sorted.

Here is my NEW 'unload' code for table Titles in Biblio.mdb (8.5K records)
Listview has columns: Title, ISBN, Year Published
If ListView is sorted by Title:
   Code is 3 times faster with re-sorting ("lvwTitles.SortKey = 1")
---------------------------------------------------
' in form declarations
Private Declare Function GetTickCount Lib "kernel32" () As Long

Private Sub Form_Unload(Cancel As Integer)
    Dim t As Long
    t = GetTickCount ' start timer
   
    If lvwTitles.Sorted = True Then
        ' sort by ISBN column
        lvwTitles.SortKey = 1 ' experiment for min. time
    End If
    lvwTitles.ListItems.Clear
   
    Debug.Print "clear in " & (GetTickCount - t) / 1000   ' report time
End Sub
---------------------------------------------------


I use Listview as main element of my data forms and I did a lot of tests with loading and unloading rows, both in VB5 and VB6. But it looks like new version of the control (SP3 or IE4.02 upgrade?) has changed a lot. In previous version this line:
    SendMessage Me.lvwTitles.hwnd, LVM_DELETEALLITEMS, 0&, 0&
was 2-3 times faster than lvwTitles.ListItems.Clear
my trick with 're-sorting' was 6 times faster

Now, LVM_DELETEALLITEMS is *the same speed* as .ListItems.Clear
're-sorting' is only 3 times faster

This is all when number of rows is 'normal' (<9000).
With 50,000 rows LockWindowUpdate made big difference.
But you should change your design and show only few hundreds of rows (or few thousands maybe)

Hi
Take a look at http://vbaccelerator.com/codelib/imalloc/objstore.htm

Cheers

Ark, I don't understand, did you mean to point to SGrid control?  Are you suggesting SGrid instead of Listview?

from http://vbaccelerator.com/codelib/sgrid/sgrid.htm 
                                                 Rows
 Control    512   1024   2048   4096   8192   16384   32767  
 
 ListView    0.18   0.36   0.74   1.49   2.78   6.23   11.0  
 FlexGrid    1.46   3.74   11.2   35.6   121   Aborted   Aborted  
 S-Grid        0.31   0.64   1.24   3.50   7.31   13.6   77  

Nope, I mean speed of direct memory access with IUnknown or IMalloc interfaces. Here is comparison. Note that ListView.ListItems is collection.
Sorry if tab not properly install, but you can look at this table at my link above.
      Collection    IUnknown    IMalloc  
 
      Add   Enum   Clear   Total    Add   Enum   Clear   Total    Add   Enum   Clear   Total  
 
 100    54   7   9   70    41   1   10   52    25   34   9   68  
 500    190   39   72   301    110   8   52   170    84   41   30   155  
 1000    383   91   224   698    201   15   94   310    157   83   69   309  
 5 000    2 184   490   4 510   7 184    1 008   73   436   1 517    843   464   298   1 605  
 10 000    4 423   1 068   28 531   34 022    1 963   168   857   2 988    1 642   935   502   3 079  
 25 000    12 352   8 947   241 672   262 971    5 601   793   2 400   8 794    4 458   2 149   1 559   8 166  

doesnt that imply he should create his own control to use those other interfaces?  i too am a bit lost but very interested by the information posted.

Ark, I think I understand what you are saying - you are only explaining why
ListView is slow.

I see IMalloc is fast.
I cannot see how to use IMalloc with Listview.

I think rbroome doesn't need super fast code/fast machine, but UI redesign.

Hi
I think that smth like this can do the job on the fly, but VB crush with this code. Need some trick to allocate/lock/unlock/free memory/heap.

  Dim LITemp As ListItems
  Dim lpLI As Long
  lpLI = ObjPtr(LV1.ListItems)
  CopyMemory LITemp, lpLI, 4&
  Set LITemp = Nothing
  CopyMemory LITemp, 0&, 4&

Cheers

Ark, you are one cool HACKER!   :-) :-) :-)
I would never try / or come to idea  for something like that.

>Need some trick to allocate/lock/unlock/free memory/heap.

Maybe use EDITBIN utility:

EDITBIN /STACK:reserve[,commit] /HEAP:reserve[,commit] mscomctllib.ocx
:-)

from http://www.vb2themax.com/Item.asp?PageID=TipBank&ID=215

http://msdn.microsoft.com/library/devprods/vs6/visualc/vccore/_core_editbin_options.htm

Whoa, I certainly generated some discussion with this one!

However, from my testing of the various solutions suggested (I haven't tried the EDITBIN and CopyMemory stuff - they sound a bit dangerous to me!) I haven't been able to get any significant performance improvements.  

I suspect that the fact that I hace already hidden the form prior to unloading is negating the benefits of LockWindowUpdate With the screen hidden, there was no redrawing being done anyway.

The DeleteAllItems is the same speed as the .clear.

The Sort trick from ameba also made no difference, however I did find that if I turned of the Sorted property of the Listview (in design mode) I did get some improvement.

So thanks for all your help. I think, as ameba suggested, I just have to avoid having such a long ListView. Now I have to convince the client why something that he had in his old DOS system (with 140,000+ records) can't be done in a new-fangled Windows system. Sigh... :-{

Hi
BTW, have you a keys in your ListItems? Simple adding keys (any) can improve performance.
Cheers

Hi
hehe, donno why, but I found this about 10 times faster:

Private Sub Form_Unload(Cancel As Integer)
  CopyMemory LV1.ListItems, 0&, 4&
End Sub

Cheers

Tested on old P120 / 64 RAM / Win95 / VB6 SP3
8.6 K listitems, 3 columns (Title, ISBN, Year Published)
Loading time:  15.5

Unloading time:
------------------------
I tested 'real' unloading time, i.e. timer is started in Form_Unload, and stopped when caller form becomes responsive:
----
' in caller form
Public start As Long ' time

Private Sub cmdTitlesSel_Click()
' show frmTitlesSel form
    Dim fsel As New frmTitlesSel
    On Error Resume Next

    fsel.Show vbModal

    Debug.Print (GetTickCount - start) / 1000 ' report time
    Set fsel = Nothing
End Sub
----
' in frmTitlesSel form
Private Sub Form_Unload(Cancel As Integer)
    tstTitles.start = GetTickCount ' start timer
   
    'If lvwTitles.Sorted = True Then
    '    lvwTitles.SortKey = 1 ' experiment for min. time
    'End If
   
    'lvwTitles.ListItems.Clear
   
    'SendMessage lvwTitles.hwnd, LVM_DELETEALLITEMS, 0&, 0&
   
    'Call CopyMemory(lvwTitles.ListItems, 0&, 4&)

End Sub
----

Unloading time for 8600 rows - RESULTS:

Unsorted    Sorted by Title            Method
 5.4              19.7                 a. No code in Form_Unload
 3.2              17.8                 b. ListItems.Clear
 2.1              19.8                 c. LVM_DELETEALLITEMS
 5.4                7.0                 d. resorting
 3.2                4.9                 e. resorting + ListItems.Clear
 2.1                6.2                 f. resorting + LVM_DELETEALLITEMS
 2.6                2.8                 g. CopyMemory

Conclusion:

When ListView is unsorted, use  LVM_DELETEALLITEMS
When ListView is sorted, use  resorting + ListItems.Clear

The best code, (for now :-)
------------------------------------------------
Private Sub Form_Unload(Cancel As Integer)
    tstTitles.start = GetTickCount  ' start timer
   
    If lvwTitles.Sorted = True Then
        lvwTitles.SortKey = 1 ' experiment for min. time
        lvwTitles.ListItems.Clear
    Else
        SendMessage lvwTitles.hwnd, LVM_DELETEALLITEMS, 0&, 0&
    End If
End Sub
------------------------------------------------
Unsorted   Sorted         Method
 2.1          4.9      resorting + ListItems.Clear + LVM_DELETEALLITEMS


CopyMemory did not qualify, because there were some side-effects:

Loading time (before using CopyMemory):  15.5
first CopyMemory call
Loading time:  20.2
second CopyMemory call
Loading time:  23.2
more CopyMemory calls
Loading time:  23.5-25.5
Also, all programs are slower, until VB IDE is restarted.

Hello, ameba
We have almost same machines (my one is P 166/32/w98/vb5:-))
I agreed that CopyMemory have same side effects so we need find some trick to avoid this. But try with more items (I used 30000) - CopyMemory effect is great! It takes almost same time ~ 3 sec in comparison with other methods which grows up.

Cheers

>try with more items

Unloading time for 17200 rows - RESULTS:

Unsorted    Sorted by Title            Method
15.8              78.9                 a. No code in Form_Unload
11.6              70.8                 b. ListItems.Clear
  5.0              84.3                 c. LVM_DELETEALLITEMS
15.8              19.1                 d. resorting
11.6              15.1                 e. resorting + ListItems.Clear
  5.0              13.2                 f. resorting + LVM_DELETEALLITEMS
  5.8                5.9                 g. CopyMemory

Loading Time: 32.0
after CopyMemory 49, then 53 seconds
after 20 times CopyMemory, my disk goes rrrrrr rrrrrrr rrrrrr rrrrr  :-)

I wonder what happens when you copy 0& into ListItems collection.
It still works after your line, e.g. .Listitems(1).Text shows correct text.

Hi
Sorry for your disk :-)
May be this help?

Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (lpDest As Any, lpSource As Any, ByVal cBytes As Long)
Private Declare Function GetProcessHeap& Lib "kernel32" ()
Private Declare Function HeapAlloc& Lib "kernel32" (ByVal hHeap As Long, ByVal dwFlags As Long, ByVal dwBytes As Long)
Private Declare Function HeapFree Lib "kernel32" (ByVal hHeap As Long, ByVal dwFlags As Long, lpMem As Any) As Long
Private Const HEAP_ZERO_MEMORY = &H8

Public Sub ClearLV(lvi As ListItems)
  Dim lHeap As Long
  lHeap = HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, 4&)
  CopyMemory ByVal lHeap, ByVal ObjPtr(lvi), 4&
  CopyMemory lvi, 0&, 4&
  HeapFree GetProcessHeap(), 0, lHeap
End Sub

Cheers

Ark
>May be this help?
No, the same side effect.

I run some additional tests - on different tables, different length of text, and got *different* results (except 'no code in Form_Unload' - which was always the slowest).  The best way is to test and measure time with the specific data.


The same mess was with clearing collections:

-in vb5 this was the fastest method:

 for i = mCol.Count to 1
    mcol.Remove 1
 loop

in vb6 this is the fastest (tested 1998, before Service Packs):

  Set mcol = Nothing


and with TreeView, this is the fastest code:
http://codeguru.earthweb.com/bbs/wt/showpost.pl?Board=vb&Number=24149&page=&view=&sb=

It uses Do...Loop to remove nodes 'one by one' (similar to AzraSound's suggestion).

There is an API method to do it without a loop (TreeView_DeleteAllItems), but it is slower (when called from VB).

Hope MS will do their 'VB garbage collection' better in the next version.

ameba,
It seems I understand the reason. This is CRAZY vb control. It can not delete all items, only one-by-one. So ALWAYS is more quicly to remove items one-by-one then with DELETEALLITEMS. Take a look at this code:

'---Bas module--
Option Explicit

Type NMHDR
    hWndFrom As Long
    idFrom As Long
    code  As Long
End Type

Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (lpDest As Any, lpSource As Any, ByVal cBytes As Long)
Private Declare Function SetWindowLong& Lib "user32" Alias "SetWindowLongA" (ByVal hwnd&, ByVal nIndex&, ByVal dwNewLong&)
Private Declare Function CallWindowProc& Lib "user32" Alias "CallWindowProcA" (ByVal lpPrevWndFunc&, ByVal hwnd&, ByVal Msg&, ByVal wParam&, ByVal lParam&)
Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" (ByVal hwnd As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long
Private Declare Function GetParent Lib "user32" (ByVal hwnd As Long) As Long
Private Declare Function DestroyWindow& Lib "user32" (ByVal hwnd As Long)


Const GWL_WNDPROC As Long = (-4&)
Const LVM_FIRST = &H1000
Const LVN_FIRST = -100
Const LVM_DELETEALLITEMS = (LVM_FIRST + 9)
Const LVN_DELETEALLITEMS = (LVN_FIRST - 4)
Const WM_NOTIFY As Long = &H4E&
Dim OldProc As Long

Public Sub ClearLV(lvw As ListView)
  If GetParent(lvw.hwnd) = 0 Then Exit Sub
  OldProc = SetWindowLong(GetParent(lvw.hwnd), GWL_WNDPROC, AddressOf WinProc)
  Call SendMessage(lvw.hwnd, LVM_DELETEALLITEMS, 0&, ByVal 0&)
  Call SetWindowLong(GetParent(lvw.hwnd), GWL_WNDPROC, OldProc)
End Sub

Public Function WinProc(ByVal hwnd As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lParam As Long) As Long
  If uMsg = WM_NOTIFY Then
     Dim nh As NMHDR
     CopyMemory nh, ByVal lParam, Len(nh)
     If nh.code = LVN_DELETEALLITEMS Then
'here it is - return True to prevent processing all LVN_DELETEITEM
        WinProc = 1
       Exit Function
     End If
  End If
  WinProc = CallWindowProc(OldProc, hwnd, uMsg, wParam, lParam)
End Function

'Place your ListView and CommandButton on form
Private Sub Command1_Click()
' Clear LV on the fly, less then 1 sec
   ClearLV LV1
'But this stupid control still hold all items!!! :-((
   Debug.Print LV1.ListItems.Count
End Sub

But now change WinProc = 0 - everything OK!
Anyway, it seems to be the most quickly method if using this code with WinProc = 0. I didn't test it with timer, but it works more quickly then simply send Deleteallitems message. Can you test it?

Cheers

PS. This is 'clear' method, no CopyMemory 'hacks'. Hope everything will be OK with your HDD :)

ameba, one more interesting thing:
add to command1_click:

   Debug.Print SendMessage(LV1.hwnd, LVM_GETITEMCOUNT, 0, 0&)

and compare with VB method
Do you understand anything???

Ark, I tested your ClearLV procedure

  Const LVM_GETITEMCOUNT = LVM_FIRST + 4
  ClearLV Me.lvwTitles
  Debug.Print lvwTitles.ListItems.Count          ' returns 8600  or  0 (for WinProc=0)
  Debug.Print SendMessage(lvwTitles.hwnd, LVM_GETITEMCOUNT, 0, 0&) ' returns 0

It proves the duality nature of ListView control
- it is SysListView32 control + some overhead to make it slow in VB  :)

>Can you test it?
for 8600 items, it is between LVM_DELETEALLITEMS and ListItems.Clear:
 Unsorted    Sorted
 3.0         20.5

Note that all times I reported are for version 6 of MS common controls (MSComctlLib)


I did few tests in version 5:
- Loading was 2 times faster (7.7 seconds)
- Unloading was ~25% faster
- ClearLV didn't work well (count is 8600 for both API and .Count methods)

>Do you understand anything???
Not much  :-)

Hi
>for 8600 items, it is between
>LVM_DELETEALLITEMS and ListItems.Clear:
> Unsorted    Sorted
> 3.0         20.5

This is for WinProc = 0?. For WinProc = 1 it should be less then 1 sec. I'll try to find where this f@#$%g VB control store this extra info and kill/erase it. Still like this simply trick with copymemory, but can not avoid side effect. Found that ListItems allocate in different places!! There are 3-4 start addresses and every ListItem (as object) takes 448 bytes. And they are mixed! For example, allocation in memory order: (First start address)ListItem(1),ListItem(2),ListItem(7),ListItem(8),ListItem(11),ListItem(12).....(Second start address)ListItem(3),ListItem(4),ListItem(5),ListItem(6),ListItem(21),ListItem(22).....etc.

Cheers

Ark,
>This is for WinProc = 0?
I tested both, and I got 3.0 and 4.0 seconds.


rbroome,
did you consider some bound grid  control (dbgrid?) to show your list?

No I hadn't considered that. I'll take a look, but generally I find bound controls too restrictive.

>I find bound controls too restrictive.
Me too.

You can also do unbound mode with DBGrid, but code is very complex...

Hello
ameba, I meant LVClear function, not unloading. I've tested LVClear with WinProc=0 on command button (not full unload, just clearing listview), and it takes less then 1 sec for 30000(!) items.
Cheesr

Ark, hi
>it takes less then 1 sec for 30000(!) items

I use data from database, and first column (field Title) text is pretty long (50-150 chars). This can be the reason why clearing is slower.

Or, maybe, my allocating memory is not continuous:

I am not doing .ListItems.Add in a continuous loop,
but I add 30 rows, then I fill array with 30 records (array will eat few memory pages), then again add rows ... so, I think my mem addresses are more spread - this should be harder to deallocate.

Another reason why clearing is slower here:

Deallocating is done by OS (VB only tells to memory menager he doesn't need pages).
Win95 is the worst OS when deallocating, and I think your Win98 is doing this better. I know NT4 is 4-10 times faster than Win95 (tested with collections 98').


Can you tell me how do you check memory addresses of Listitems. I would like to see if my items are really spread in memory.
Is it 'ObjPtr(Listitem)'?

And, are you using this code to fill your ListView?

>    For i = 1 To 30000
>      With .ListItems.Add(, , "Item " & CStr(i))
>        .SubItems(1) = "Subitem 1"
>        .SubItems(2) = "Subitem 2"
>      End With
>    Next

Thanks

Hi,
To load, I used this (faster) code:

    With LV1.ListItems
        For i = 1 To numrows
           .Add , , "Item " & CStr(i)
        Next
    End With

Items Time

10000     4,697    Load
10000   20,749   .Clear
10000     1,404    LVM_Del
10000     1,889    ClearLV WinProc = 0
10000     0,065    ClearLV WinProc = 1

20000     9,669    Load
20000   79,649    .Clear
20000     2,898    LVM_Del
20000     3,954    ClearLV WinProc = 0
20000     0,178    ClearLV WinProc = 1

30000     14,805   Load
30000   174,338   .Clear
30000       4,406    LVM_Del
30000       6,055    ClearLV WinProc = 0
30000       0,374    ClearLV WinProc = 1


ClearLV with WinProc = 1  had side effects ('not responding' both in IDE and compiled) when ClearLV is used few times.  :-(

My conclusion after this discussion:
Test misc. unloading methods with your data set.

rbroome, thanks for asking this question - it caused 'invalidate' event and forced 'retest' method on my lv_knowledge object.  :-)

Ark, thanks for your help and discussion, I posted question for you:
https://www.experts-exchange.com/jsp/qShow.jsp?ta=visualbasic&qid=10854321 

<ameba goes testing flexgrid loading/clearing, I noticed it is now faster (two years ago it was 2 times slower than listview)>

Maybe you should dynamicly add and remove items from the listview as the user scrolls around. Doing this will keep the ammount of data in the list to a very small ammount and you can store all the data outside of the listview to allow multi directional scrolling.

Just an Idea....

All the discussion on this has been very revealing, although as it turned out for my application it didn't really help. As I load the listview before the screen is displayed, and unload it after, none of the tweaks to the refreshing had any effect. I ended up restricting the list by forcing the user to enter some characters before retrieving the records, hence limiting the amount of data being put into the listview.

Can I somehow give ameba and ark half points each, as they have both contributed to my VB knowledge, even if not directly applicable to this specific instance.

>Can I somehow give ameba and ark half points each
Thanks, but this is not trivial and requires some action from Community support.  I also feel we did not deserve the points, and my statement:
>Yes, I know faster way(s).
was for some old listview  :-)

I suggest you simply delete the question.  I saved link for my reference, though.

ark, are you happy with this?

rbroome.. drop a ZERO point Question in the Community Support area instructing them on how you would like the points split:

https://www.experts-exchange.com/Customer_Service/Experts_Exchange/

In the question.. copy/paste this Question's hyperlink into the subject:

https://www.experts-exchange.com/jsp/qShow.jsp?ta=visualbasic&qid=10753321 

And the CS people will take it from there.. <smile>.

Hello everyone,

Reducing points to 350 to allow for split.

rbroome - You can now accept one of Ark's comments in this thread as an answer. To award Expert ameba, you can create a new question in this topic area with a title of 'For ameba -- 10753321' for 150 points.

Remember, the Accept Comment as Answer button is in the header of the comment.

For your convenience, you can use this link to create the new question (right click and open in new window):
https://www.experts-exchange.com/bin/NewQForm?ta=31

Be sure to come back and let the Expert know about the new question once you have posted it.

darinw
Community Support