Preloading a large text file.

Posted on 2006-03-20
Last Modified: 2013-11-13
I read through text files one by one, I do this to try and prevent the buffer from becoming so overwelmed that the application becomes a pain in the arse to manage memory wise.  The current method I use is:

I load a list first, to count the number of items in it:

Dim lngCount As Long
Dim strCurrItem As String

Open StrDicList For Input As #1

    Do Until EOF(1)
        Line Input #1, strCurrItem
        lngCount = lngCount + 1
        Label12.Caption = "Total Loaded: " & lngCount
Close #1

Then I open it so the program may process the data 1 by 1:

Open StrDicList For Input As #1
Exit Function

Then this is the code I use to go through one by one:

Do Until EOF(1)
Line Input #1, StrCurrWord
debug.print StrCurrWord
exit sub

Now, what I am wondering, is there any possible way to load this faster?  Like, to count the number of items in a larger text file.  Because it seems to take quite a bit.  I would like to be able to load a text file as fast as richtextbox control is capable of.  I do not use richtextbox control because it stores the entire list into a array and the program becomes extremely bloated.  Does anyone know of any possible multithreaded ability to open a text file with x amount of threads so you're counting like 50 words every X amount of time instead of 1.

Any ideas would be greatful.
Question by:moorer2k
    LVL 17

    Accepted Solution

    What size files are you dealing with? If under 50mb, just buffer it all.

    >>Because it seems to take quite a bit
    DoEvents on every loop will slow it down a lot and it's not required. You should only call it, at minimum, every 100ms. It should give enough time to display any form refreshes and interactions to process.

    In this situation, you'll either have to sacrifice memory or speed. I'd rather sacrifice memory as most modern systems have quite a bit, and page-files are usually large.

    Here's an example below (same concept as RichEdit). If file-sizes exceed 50mb, you'll probably wait a different approach.

    Option Explicit

    Private m_fBuff() As String
    Private Sub Form_Load()
        If (FileLoad("c:\test.txt") = True) Then
            Call MsgBox("Lines: " & CStr(UBound(m_fBuff) + 1), vbInformation)
        End If
    End Sub
    Private Function FileLoad(ByVal sFile As String) As Boolean
        Dim intFF As Integer
        Dim strBuff As String
        intFF = FreeFile
        If (Dir$(sFile, vbNormal Or vbHidden) <> vbNullString) Then
            Open sFile For Binary As intFF
                strBuff = Space$(LOF(intFF))
                Get #intFF, 1, strBuff
                m_fBuff = Split(strBuff, vbCrLf)
                FileLoad = True
            Close intFF
        End If
    End Function


    Author Comment

    Oh wow, I did not know that DoEvents slowed down the process so much!  I removed it just for testing, and it loaded it instantly!  That fixed my functions totally, I will add it to do it every 100 MS or so, thanks!

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    IT, Stop Being Called Into Every Meeting

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    Introduction In a recent article ( for the Excel community, I showed an improved version of the Excel Concatenate() function.  While writing that article I realized that no o…
    Most everyone who has done any programming in VB6 knows that you can do something in code like Debug.Print MyVar and that when the program runs from the IDE, the value of MyVar will be displayed in the Immediate Window. Less well known is Debug.Asse…
    Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…
    Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    10 Experts available now in Live!

    Get 1:1 Help Now