Solved

Followup to previous recursive folder loop - showing thumbnails - pt 6

Posted on 2006-10-29
26
195 Views
Last Modified: 2008-02-01
http://www.experts-exchange.com/Programming/Programming_Languages/Dot_Net/VB_DOT_NET/Q_22036099.html is the last question in this 'series'.

I'd like to speed up the processing of my images, which will 'average' about 9000 subfolders, and potentially over 150,000 images...
0
Comment
Question by:sirbounty
  • 14
  • 12
26 Comments
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17830274
Hi sirbounty;

Try these changes in the effort to speed up the displaying of the images to the user and if you would like to try it this is what you need to do. This change uses a worker thread to get all the files names and updates the UserImagesNames array by the amount of DisplayCount. So every full page of images will be available to the user.

Add this line at class level

    ' The Worker thread
    Private WithEvents BgWorker As New System.ComponentModel.BackgroundWorker

Remove these two lines from the Form Load event

        process()
        DisplayImages()

And add this line of code

        ' Start the background worker thread. Thread runs in DoWork event.
        BgWorker.RunWorkerAsync()

Remove the Complete Private Sub DisplayImages() subroutine and replace it with the following event handler

    ' Start the Asynchronous file search. The background thread does its work from
    ' this event.
    Private Sub BgWorker_DoWork(ByVal sender As Object, _
        ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BgWorker.DoWork

        EndWorkerThreadToolStripMenuItem.Enabled = True

        ' Holds the filepath to all the JPG.LST files in question
        Dim JPGFiles() As String
        ' Temp storage for image file names
        Dim ImageFileNames As New ArrayList
        Dim FirstTimeThrough As Boolean = True

        ' Get the string file names of all the JPG.LST files
        JPGFiles = Directory.GetFiles(strFile, "JPG.LST", SearchOption.AllDirectories)
        ' Fill the UserImagesNames array with all the image names
        For Each file As String In JPGFiles
            Using sr As New StreamReader(file)
                Dim input As String = sr.ReadToEnd()
                ImageFileNames.AddRange(input.Split(CrLf, StringSplitOptions.RemoveEmptyEntries))
            End Using
            If ImageFileNames.Count > DisplayCount - 1 Then
                UserImagesNames.AddRange(ImageFileNames.GetRange(0, DisplayCount - 1))
                ImageFileNames.RemoveRange(0, DisplayCount)
                If FirstTimeThrough Then
                    FirstTimeThrough = False
                    BgWorker.ReportProgress(0)
                End If
            End If
        Next

        If ImageFileNames.Count > 0 Then
            UserImagesNames.AddRange(ImageFileNames.GetRange(0, ImageFileNames.Count))
        End If

    End Sub

The EndWorkerThreadToolStripMenuItem assumes that you will add a way to stop the worker thread from running before it has completed because you will be ending the program. I placed this on the tool strip by that name and set its initial Enabled property to false. This is also true for the control name in the BgWorker_RunWorkerCompleted coming up next.

Add the following three event handlers to your code

    ' This is called only once just after the first DisplayCount images are available
    ' to the user
    Private Sub BgWorker_ProgressChanged(ByVal sender As Object, _
        ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
        Handles BgWorker.ProgressChanged

        DisplayImages()

    End Sub

    ' The background thread calls this event just before it reaches the End Sub
    ' of the DoWork event. It is OK to access user controls in the UI from this event.
    Private Sub BgWorker_RunWorkerCompleted(ByVal sender As Object, _
        ByVal e As System.ComponentModel.RunWorkerCompletedEventArgs) _
        Handles BgWorker.RunWorkerCompleted

        EndWorkerThreadToolStripMenuItem.Enabled = False
        BgWorker.Dispose()

    End Sub

    Private Sub EndWorkerThreadToolStripMenuItem_Click(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles EndWorkerThreadToolStripMenuItem.Click

        BgWorker.CancelAsync()

    End Sub


Let me know how this works out for you.

Fernando
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17830831
"Remove the Complete Private Sub DisplayImages() subroutine and replace it with the following event handler"

But we're still calling it here...

Private Sub BgWorker_ProgressChanged(ByVal sender As Object, _
        ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
        Handles BgWorker.ProgressChanged

        DisplayImages()

??  

What should be in its place?
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17830958
Hi sirbounty;

Here I go again. I think I need to make an appointment with my doctor to find out what is going on in my brain, but I am afraid that he will tell me that there is nothing up there. LOL.

This statement :

Remove the Complete Private Sub DisplayImages() subroutine and replace it with the following event handler

Should have been read as:

Remove the Complete process subroutine and replace it with the following event handler

So let me try it this way

Remove this code:

    Private Sub process()

        Dim JPGFiles() As String

        JPGFiles = Directory.GetFiles(strFile, _
             "JPG.LST", SearchOption.AllDirectories)

        For Each file As String In JPGFiles
            Using sr As New StreamReader(file)
                Dim input As String = sr.ReadToEnd()
                UserImagesNames.AddRange(input.Split(CrLf, _
                     StringSplitOptions.RemoveEmptyEntries))
            End Using
        Next

    End Sub

And replace it with this code.

    ' Start the Asynchronous file search. The background thread does its work from
    ' this event.
    Private Sub BgWorker_DoWork(ByVal sender As Object, _
        ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BgWorker.DoWork

        EndWorkerThreadToolStripMenuItem.Enabled = True

        ' Holds the filepath to all the JPG.LST files in question
        Dim JPGFiles() As String
        ' Temp storage for image file names
        Dim ImageFileNames As New ArrayList
        Dim FirstTimeThrough As Boolean = True

        ' Get the string file names of all the JPG.LST files
        JPGFiles = Directory.GetFiles(strFile, "JPG.LST", SearchOption.AllDirectories)
        ' Fill the UserImagesNames array with all the image names
        For Each file As String In JPGFiles
            Using sr As New StreamReader(file)
                Dim input As String = sr.ReadToEnd()
                ImageFileNames.AddRange(input.Split(CrLf, StringSplitOptions.RemoveEmptyEntries))
            End Using
            If ImageFileNames.Count > DisplayCount - 1 Then
                UserImagesNames.AddRange(ImageFileNames.GetRange(0, DisplayCount - 1))
                ImageFileNames.RemoveRange(0, DisplayCount)
                If FirstTimeThrough Then
                    FirstTimeThrough = False
                    BgWorker.ReportProgress(0)
                End If
            End If
        Next

        If ImageFileNames.Count > 0 Then
            UserImagesNames.AddRange(ImageFileNames.GetRange(0, ImageFileNames.Count))
        End If

    End Sub

And do not touch the DisplayImages subroutine.

OK, I think I got it right this time.

Fernando
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17831019
Presumably I just won't see anything until it's grabbed a pageful of images?
I don't really like the bgworker trick, primarily cause I don't understand it perhaps, but it doesn't seem too 'chatty' when it's running either - thus, how do you debug it? :)

I just kicked it off - will wait a half hour and see if anything appears...thanx!
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17831098
To your question, "Presumably I just won't see anything until it's grabbed a pageful of images?", The BgWorker_DoWork code get created and called from the Form Load event and starts to collect the names of the image files when it has gotten DisplayCount number of file names it transfer the names to the UserImagesNames array. When BgWorker_DoWork hits the first DisplayCount of file names it raises the event ProgressChanged which executes this event handler which displays each of the images one at a time in the FlowLayoutPanel.

    Private Sub BgWorker_ProgressChanged(ByVal sender As Object, _
        ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
        Handles BgWorker.ProgressChanged

        DisplayImages()

    End Sub

To your question, "thus, how do you debug it?". You can place Console.WriteLines in the code, write to a log file, or set breakpoints in the code.

It should not take very log until you see the first set of images. So if you do not see anything in about 10 sec or so please post the file to the EE web sit so that I may look at it.

Fernando
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17831223
Nothing yet... :(
I'll upload the project.
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17831256
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17831528
Hi sirbounty;

Place these two lines in the Form Load event just before the statement, BgWorker.RunWorkerAsync().

        ' Enable ProgressChanged and Cancellation mothed
        BgWorker.WorkerReportsProgress = True
        BgWorker.WorkerSupportsCancellation = True

Also you need these events handlers in order for the solution to work.

    ' This is called only once just after the first DisplayCount images are available
    ' to the user
    Private Sub BgWorker_ProgressChanged(ByVal sender As Object, _
        ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
        Handles BgWorker.ProgressChanged

        DisplayImages()

    End Sub

    ' The background thread calls this event just before it reaches the End Sub
    ' of the DoWork event. It is OK to access user controls in the UI from this event.
    Private Sub BgWorker_RunWorkerCompleted(ByVal sender As Object, _
        ByVal e As System.ComponentModel.RunWorkerCompletedEventArgs) _
        Handles BgWorker.RunWorkerCompleted

        EndWorkerThreadToolStripMenuItem.Enabled = False
        BgWorker.Dispose()

    End Sub

    Private Sub EndWorkerThreadToolStripMenuItem_Click(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles EndWorkerThreadToolStripMenuItem.Click

        BgWorker.CancelAsync()

    End Sub

Fernando
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17831549
Been running for a couple minutes - nothing yet... 8 |

I paused processing and it appears to be at the point where it's building the JPG array - that usually takes a while, unless it somehow breaks out of that when DisplayCount is full?
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17831574
Again you should have seen some images almost immediately. This code will read file names from JPG.LST files and when it has collected DisplayCount number of files which might be 1, 2 or 3 JPG.LST files will start showing up on the display.  
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17831584
Hmm - let me try restarting it then...I till have nothing after over 10 minutes...
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17831611
Did you add the code I just posted?
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17831644
Still nothing...  
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17831662
I will take a another look at the code you sent in the morning to see if I missed anything.
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17834336
Hi sirbounty;

I went over what you upload to the web site and I made it what it should be. Compare it to what you have for frmViewe. I have run this code on my system and it does work. You should be able to copy your current project to another location and replace the frmViewer.vb file with the one I posted below.

https://filedb.experts-exchange.com/incoming/ee-stuff/1271-frmViewer.zip

I noticed that you added the EndWorkerThreadToolStripMenuItem in the Images menu item on the main menu. I would either enable Images so that you have access to the Stop Processing menu item or place it somewhere else.

Fernando
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17843555
"I noticed that you added the EndWorkerThreadToolStripMenuItem in the Images menu item on the main menu. I would either enable Images so that you have access to the Stop Processing menu item or place it somewhere else."

I'm not sure what you mean by that?  Are you referring to the "Save Position" item? I haven't really fully developed that - it was more of an attempt to save the file that was currently being read, along with the position in that file, if the user decided to close out the application altogether, so they could pick back up where they left off, rather than running the full gamut a second time...

I've been running off the same instance from the other day - and have noticed that the images seem to be loading slower than they were (averaged about per second initially).  It could be simply due to the bandwidth of the server that's being loaded from...not sure.  But if it's a decrease of available resources, I may eventually want to have that Save Position item available so that I can start fresh (just something to think about - certainly a new question if it becomes too much of an issue).

I will download the latest and let you know - thanx!
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17843664
appears to be a logon problem (for me anyway) - can't get to ee-stuff today... :(
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17848637
can't logon today either...arg
Can you get back in to ee-stuff?
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17877427
I sent an e-mail to them on the 2nd and it seems to be working today the 5th.
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17880251
Okay, I grabbed the file this morning, but it seems to be responding the same way...

In looking at the code, wouldn't this line need to completely finish (i.e. reading all 9000 subfolders) before proceeding to the next line?

 JPGFiles = Directory.GetFiles(strFile, "JPG.LST", SearchOption.AllDirectories)
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17881509
To your question, “In looking at the code, wouldn't this line need to completely finish (i.e. reading all 9000 subfolders) before proceeding to the next line?”. It looks at the system file table and locates the 9000 file names so in answer to your question Yes.

I put together some test code to check the speed of the above code in question. The time it take to complete this line of code, JPGFiles = Directory.GetFiles(strFile, "JPG.LST", SearchOption.AllDirectories), The results of that test will be displayed in the forms title bar in the following format Minutes:Seconds:Milliseconds.  

    ‘ Place this at class level, outside of any function or subroutine.
    Dim strTime As String = ""      ' Test Speed



    ‘ In this subroutine add all the lines that have the comment “Test Speed”

    Private Sub BgWorker_DoWork(ByVal sender As Object, _
        ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BgWorker.DoWork

        EndWorkerThreadToolStripMenuItem.Enabled = True

        ' Holds the filepath to all the JPG.LST files in question
        Dim JPGFiles() As String
        ' Temp storage for image file names
        Dim ImageFileNames As New ArrayList
        Dim FirstTimeThrough As Boolean = True

        Dim sw As New Stopwatch()' Test Speed
        sw.Stop()      ' Test Speed
        sw.Reset()      ' Test Speed
        sw.Start()      ' Test Speed

        ' Get the string file names of all the JPG.LST files
        JPGFiles = Directory.GetFiles(strFile, "JPG.LST", SearchOption.AllDirectories)
        ' Fill the UserImagesNames array with all the image names

        sw.Stop()      ' Test Speed
        strTime = sw.Elapsed.Minutes.ToString & ":" & sw.Elapsed.Seconds.ToString & ":" & _
            sw.Elapsed.Milliseconds.ToString      ' Test Speed
        BgWorker.ReportProgress(0)      ' Test Speed

        For Each file As String In JPGFiles
            Using sr As New StreamReader(file)
                Dim input As String = sr.ReadToEnd()
                ImageFileNames.AddRange(input.Split(CrLf, StringSplitOptions.RemoveEmptyEntries))
            End Using
            If ImageFileNames.Count > DisplayCount - 1 Then
                UserImagesNames.AddRange(ImageFileNames.GetRange(0, DisplayCount - 1))
                ImageFileNames.RemoveRange(0, DisplayCount)
                If FirstTimeThrough Then
                    FirstTimeThrough = False
                    BgWorker.ReportProgress(0)
                End If
            End If
        Next

        If ImageFileNames.Count > 0 Then
            UserImagesNames.AddRange(ImageFileNames.GetRange(0, ImageFileNames.Count))
        End If

    End Sub


    ‘ In this subroutine add all the lines that have the comment “Test Speed”

    Private Sub BgWorker_ProgressChanged(ByVal sender As Object, _
        ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
        Handles BgWorker.ProgressChanged

        If UserImagesNames.Count > 0 Then      ' Test Speed
            DisplayImages()
        End If      ' Test Speed
        Me.Text = strTime      ' Test Speed
        Me.Refresh()      ' Test Speed

    End Sub


Let me know what results you get I would be interested.

Fernando
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17913412
I apologize for the delay.
I've got this running now.  Will let you know shortly...well, when it's done anyway. :)
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17913568
28:56:23
0
 
LVL 62

Accepted Solution

by:
Fernando Soto earned 500 total points
ID: 17925331
Hi sirbounty;

That time is just unacceptable I just can't understand why it is taking so long.

I made the following changes to see if this version will work any better.

    ' New Class level variables added
    Private CallOnce As Boolean = True
    Private CurrentDispCount As Integer = 0

New version of  BgWorker_DoWork subroutine

    Private Sub BgWorker_DoWork(ByVal sender As Object, _
        ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BgWorker.DoWork

        EndWorkerThreadToolStripMenuItem.Enabled = True
        QueueImageFiles(strFile)

    End Sub

New subroutine added.

    Private Sub QueueImageFiles(ByVal dir As String)

        Dim JPGFile As String = dir & "\JPG.LST"
        If My.Computer.FileSystem.FileExists(JPGFile) Then
            Using sr As New StreamReader(JPGFile)
                Dim input As String = sr.ReadToEnd()
                UserImagesNames.AddRange(input.Split(CrLf, StringSplitOptions.RemoveEmptyEntries))
            End Using
            If CallOnce AndAlso UserImagesNames.Count > 0 Then
                BgWorker.ReportProgress(0)
                CallOnce = False
            End If
        End If

        For Each NextDir As String In My.Computer.FileSystem.GetDirectories(dir)
            QueueImageFiles(NextDir)
        Next

    End Sub

New version of BgWorker_ProgressChanged subroutine

    Private Sub BgWorker_ProgressChanged(ByVal sender As Object, _
        ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
        Handles BgWorker.ProgressChanged

        Timer1.Enabled = True
        Timer1_Tick(Nothing, Nothing)

    End Sub

New version of DisplayImages subroutine

    Private Sub DisplayImages()

        If EndWorkerThreadToolStripMenuItem.Enabled = False Then Timer1.Enabled = False
        If NextImage >= UserImagesNames.Count Then Return

        ' Control type to be place in the FlowLayoutPanel
        Dim pb As PictureBox

        If NextImage Mod DisplayCount = 0 Then
            For Each Ctl As Control In FlowLayoutPanel1.Controls
                Dim pbox As PictureBox = CType(Ctl, PictureBox)
                RemoveHandler pbox.DoubleClick, AddressOf DisplaySingleImage
            Next
            ToolTip1.RemoveAll()
            FlowLayoutPanel1.Controls.Clear()
        End If

        ' Get and dispplay the images
        For idx As Integer = 0 To DisplayCount - 1
            pb = New PictureBox
            ' Add the DoubleClick event handler to the picture box
            AddHandler pb.DoubleClick, AddressOf DisplaySingleImage
            pb.SizeMode = PictureBoxSizeMode.AutoSize
            pb.Margin = New System.Windows.Forms.Padding(1, 1, 1, 1)
            pb.Size = New Size(imageWidth, imageHeight)
            pb.SizeMode = PictureBoxSizeMode.StretchImage
            Try
                Dim bMap As New Bitmap(UserImagesNames(NextImage).ToString)
                Dim myThumbnail As Image = bMap.GetThumbnailImage( _
                    imageWidth, imageHeight, cbAbort, IntPtr.Zero)
                pb.Image = myThumbnail
            Catch ex As Exception
                ' Write the name of the file out to a log file
                'pb.Image = Image.FromFile("\\Midp-sfs-002\Techeng\Audpull\Retrieve\NoLoad.jpg")
                Console.WriteLine("Image file not displayed " & UserImagesNames(NextImage).ToString)
            End Try
            pb.Tag = UserImagesNames(NextImage).ToString()
            ToolTip1.SetToolTip(pb, UserImagesNames(NextImage).ToString())
            pb.Show()
            FlowLayoutPanel1.Controls.Add(pb)
            CurrentDispCount += 1
            NextImage += 1
            Application.DoEvents()
            If CurrentDispCount = DisplayCount OrElse _
                NextImage >= UserImagesNames.Count Then Exit For
        Next

        ImagesToolStripMenuItem.Enabled = True

    End Sub

Added a Timer to this form called timer1. I set the Interval property to 5000 which is every 5 seconds.

    Private Sub Timer1_Tick(ByVal sender As Object, _
        ByVal e As System.EventArgs) Handles Timer1.Tick

        If CurrentDispCount < DisplayCount Then
            DisplayImages()
        End If

    End Sub

New version of NextGroupToolStripMenuItem_Click event handler.

    Private Sub NextGroupToolStripMenuItem_Click(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles NextGroupToolStripMenuItem.Click

        ImagesToolStripMenuItem.Enabled = False
        If Timer1.Enabled = True Then
            CurrentDispCount = 0
        End If
        DisplayImages()

    End Sub


I added the frmViewer.vb file to the EE web site that I used for testing just in case I missed something so that you can download it. The link is

    https://filedb.experts-exchange.com/incoming/ee-stuff/1362-frmViewer.zip


Fernando
0
 
LVL 67

Author Comment

by:sirbounty
ID: 17937473
Got a chance to implement this today.
Here's what I see...not sure yet 'why', as I haven't stepped through it yet.
After approximately 25 seconds, the images started appearing at the 15th one on the first line.  My "Unable to load" image didn't appear in the first 14 (and I haven't checked yet to see if that code simply not there in this latest version).
Plus, I got two lines too many (scrollbars appeared on the right) - but I imagine I can correct that myself.
Lastly, when the Next Images menuitem became available, clicking it almost immediately retrieved the next images.
Awesome work sir - I'm very impressed!
Thank you for your dedication to helping here.  Hopefully when I go over the code, I'll understand it enough to actually learn something. :^)
Again, I appreciate your help very much!
0
 
LVL 62

Expert Comment

by:Fernando Soto
ID: 17938277
Always glad to be able to help. Let me know what you find. ;=)
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

The ECB site provides FX rates for major currencies since its inception in 1999 in the form of an XML feed. The files have the following format (reducted for brevity) (CODE) There are three files available HERE (http://www.ecb.europa.eu/stats/exch…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now