[Webinar] Learn how to a build a cloud-first strategyRegister Now


Preloading a large text file.

Posted on 2006-03-20
Medium Priority
Last Modified: 2013-11-13
I read through text files one by one, I do this to try and prevent the buffer from becoming so overwelmed that the application becomes a pain in the arse to manage memory wise.  The current method I use is:

I load a list first, to count the number of items in it:

Dim lngCount As Long
Dim strCurrItem As String

Open StrDicList For Input As #1

    Do Until EOF(1)
        Line Input #1, strCurrItem
        lngCount = lngCount + 1
        Label12.Caption = "Total Loaded: " & lngCount
Close #1

Then I open it so the program may process the data 1 by 1:

Open StrDicList For Input As #1
Exit Function

Then this is the code I use to go through one by one:

Do Until EOF(1)
Line Input #1, StrCurrWord
debug.print StrCurrWord
exit sub

Now, what I am wondering, is there any possible way to load this faster?  Like, to count the number of items in a larger text file.  Because it seems to take quite a bit.  I would like to be able to load a text file as fast as richtextbox control is capable of.  I do not use richtextbox control because it stores the entire list into a array and the program becomes extremely bloated.  Does anyone know of any possible multithreaded ability to open a text file with x amount of threads so you're counting like 50 words every X amount of time instead of 1.

Any ideas would be greatful.
Question by:moorer2k
LVL 17

Accepted Solution

zzzzzooc earned 200 total points
ID: 16244637
What size files are you dealing with? If under 50mb, just buffer it all.

>>Because it seems to take quite a bit
DoEvents on every loop will slow it down a lot and it's not required. You should only call it, at minimum, every 100ms. It should give enough time to display any form refreshes and interactions to process.

In this situation, you'll either have to sacrifice memory or speed. I'd rather sacrifice memory as most modern systems have quite a bit, and page-files are usually large.

Here's an example below (same concept as RichEdit). If file-sizes exceed 50mb, you'll probably wait a different approach.

Option Explicit

Private m_fBuff() As String
Private Sub Form_Load()
    If (FileLoad("c:\test.txt") = True) Then
        Call MsgBox("Lines: " & CStr(UBound(m_fBuff) + 1), vbInformation)
    End If
End Sub
Private Function FileLoad(ByVal sFile As String) As Boolean
    Dim intFF As Integer
    Dim strBuff As String
    intFF = FreeFile
    If (Dir$(sFile, vbNormal Or vbHidden) <> vbNullString) Then
        Open sFile For Binary As intFF
            strBuff = Space$(LOF(intFF))
            Get #intFF, 1, strBuff
            m_fBuff = Split(strBuff, vbCrLf)
            FileLoad = True
        Close intFF
    End If
End Function


Author Comment

ID: 16244692
Oh wow, I did not know that DoEvents slowed down the process so much!  I removed it just for testing, and it loaded it instantly!  That fixed my functions totally, I will add it to do it every 100 MS or so, thanks!

Featured Post

Prep for the ITIL® Foundation Certification Exam

December’s Course of the Month is now available! Enroll to learn ITIL® Foundation best practices for delivering IT services effectively and efficiently.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever wanted to restrict the users input in a textbox to numbers, and while doing that make sure that they can't 'cheat' by pasting in non-numeric text? Of course you can do that with code you write yourself but it's tedious and error-prone …
When designing a form there are several BorderStyles to choose from, all of which can be classified as either 'Fixed' or 'Sizable' and I'd guess that 'Fixed Single' or one of the other fixed types is the most popular choice. I assume it's the most p…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…
This lesson covers basic error handling code in Microsoft Excel using VBA. This is the first lesson in a 3-part series that uses code to loop through an Excel spreadsheet in VBA and then fix errors, taking advantage of error handling code. This l…
Suggested Courses
Course of the Month20 days, 13 hours left to enroll

865 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question