Preloading a large text file.

I read through text files one by one, I do this to try and prevent the buffer from becoming so overwelmed that the application becomes a pain in the arse to manage memory wise.  The current method I use is:

I load a list first, to count the number of items in it:

Dim lngCount As Long
Dim strCurrItem As String

Open StrDicList For Input As #1

    Do Until EOF(1)
   
        Line Input #1, strCurrItem
        lngCount = lngCount + 1
        DoEvents
        Label12.Caption = "Total Loaded: " & lngCount
    Loop
   
Close #1

Then I open it so the program may process the data 1 by 1:

Open StrDicList For Input As #1
Exit Function

Then this is the code I use to go through one by one:

Do Until EOF(1)
Line Input #1, StrCurrWord
debug.print StrCurrWord
exit sub
loop

Now, what I am wondering, is there any possible way to load this faster?  Like, to count the number of items in a larger text file.  Because it seems to take quite a bit.  I would like to be able to load a text file as fast as richtextbox control is capable of.  I do not use richtextbox control because it stores the entire list into a array and the program becomes extremely bloated.  Does anyone know of any possible multithreaded ability to open a text file with x amount of threads so you're counting like 50 words every X amount of time instead of 1.

Any ideas would be greatful.
moorer2kAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

zzzzzoocCommented:
What size files are you dealing with? If under 50mb, just buffer it all.

>>Because it seems to take quite a bit
DoEvents on every loop will slow it down a lot and it's not required. You should only call it, at minimum, every 100ms. It should give enough time to display any form refreshes and interactions to process.

In this situation, you'll either have to sacrifice memory or speed. I'd rather sacrifice memory as most modern systems have quite a bit, and page-files are usually large.

Here's an example below (same concept as RichEdit). If file-sizes exceed 50mb, you'll probably wait a different approach.

Form1:
---------------
Option Explicit

Private m_fBuff() As String
Private Sub Form_Load()
    If (FileLoad("c:\test.txt") = True) Then
        Call MsgBox("Lines: " & CStr(UBound(m_fBuff) + 1), vbInformation)
    End If
End Sub
Private Function FileLoad(ByVal sFile As String) As Boolean
    Dim intFF As Integer
    Dim strBuff As String
    intFF = FreeFile
    If (Dir$(sFile, vbNormal Or vbHidden) <> vbNullString) Then
        Open sFile For Binary As intFF
            strBuff = Space$(LOF(intFF))
            Get #intFF, 1, strBuff
            m_fBuff = Split(strBuff, vbCrLf)
            FileLoad = True
        Close intFF
    End If
End Function

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
moorer2kAuthor Commented:
Oh wow, I did not know that DoEvents slowed down the process so much!  I removed it just for testing, and it loaded it instantly!  That fixed my functions totally, I will add it to do it every 100 MS or so, thanks!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Installation

From novice to tech pro — start learning today.