Preloading a large text file.

I read through text files one by one, I do this to try and prevent the buffer from becoming so overwelmed that the application becomes a pain in the arse to manage memory wise.  The current method I use is:

I load a list first, to count the number of items in it:

Dim lngCount As Long
Dim strCurrItem As String

Open StrDicList For Input As #1

    Do Until EOF(1)
   
        Line Input #1, strCurrItem
        lngCount = lngCount + 1
        DoEvents
        Label12.Caption = "Total Loaded: " & lngCount
    Loop
   
Close #1

Then I open it so the program may process the data 1 by 1:

Open StrDicList For Input As #1
Exit Function

Then this is the code I use to go through one by one:

Do Until EOF(1)
Line Input #1, StrCurrWord
debug.print StrCurrWord
exit sub
loop

Now, what I am wondering, is there any possible way to load this faster?  Like, to count the number of items in a larger text file.  Because it seems to take quite a bit.  I would like to be able to load a text file as fast as richtextbox control is capable of.  I do not use richtextbox control because it stores the entire list into a array and the program becomes extremely bloated.  Does anyone know of any possible multithreaded ability to open a text file with x amount of threads so you're counting like 50 words every X amount of time instead of 1.

Any ideas would be greatful.
moorer2kAsked:
Who is Participating?
 
zzzzzoocCommented:
What size files are you dealing with? If under 50mb, just buffer it all.

>>Because it seems to take quite a bit
DoEvents on every loop will slow it down a lot and it's not required. You should only call it, at minimum, every 100ms. It should give enough time to display any form refreshes and interactions to process.

In this situation, you'll either have to sacrifice memory or speed. I'd rather sacrifice memory as most modern systems have quite a bit, and page-files are usually large.

Here's an example below (same concept as RichEdit). If file-sizes exceed 50mb, you'll probably wait a different approach.

Form1:
---------------
Option Explicit

Private m_fBuff() As String
Private Sub Form_Load()
    If (FileLoad("c:\test.txt") = True) Then
        Call MsgBox("Lines: " & CStr(UBound(m_fBuff) + 1), vbInformation)
    End If
End Sub
Private Function FileLoad(ByVal sFile As String) As Boolean
    Dim intFF As Integer
    Dim strBuff As String
    intFF = FreeFile
    If (Dir$(sFile, vbNormal Or vbHidden) <> vbNullString) Then
        Open sFile For Binary As intFF
            strBuff = Space$(LOF(intFF))
            Get #intFF, 1, strBuff
            m_fBuff = Split(strBuff, vbCrLf)
            FileLoad = True
        Close intFF
    End If
End Function

0
 
moorer2kAuthor Commented:
Oh wow, I did not know that DoEvents slowed down the process so much!  I removed it just for testing, and it loaded it instantly!  That fixed my functions totally, I will add it to do it every 100 MS or so, thanks!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.