We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you two Citrix podcasts. Learn about 2020 trends and get answers to your biggest Citrix questions!Listen Now

x

Preloading a large text file.

Medium Priority
413 Views
Last Modified: 2013-11-13
I read through text files one by one, I do this to try and prevent the buffer from becoming so overwelmed that the application becomes a pain in the arse to manage memory wise.  The current method I use is:

I load a list first, to count the number of items in it:

Dim lngCount As Long
Dim strCurrItem As String

Open StrDicList For Input As #1

    Do Until EOF(1)
   
        Line Input #1, strCurrItem
        lngCount = lngCount + 1
        DoEvents
        Label12.Caption = "Total Loaded: " & lngCount
    Loop
   
Close #1

Then I open it so the program may process the data 1 by 1:

Open StrDicList For Input As #1
Exit Function

Then this is the code I use to go through one by one:

Do Until EOF(1)
Line Input #1, StrCurrWord
debug.print StrCurrWord
exit sub
loop

Now, what I am wondering, is there any possible way to load this faster?  Like, to count the number of items in a larger text file.  Because it seems to take quite a bit.  I would like to be able to load a text file as fast as richtextbox control is capable of.  I do not use richtextbox control because it stores the entire list into a array and the program becomes extremely bloated.  Does anyone know of any possible multithreaded ability to open a text file with x amount of threads so you're counting like 50 words every X amount of time instead of 1.

Any ideas would be greatful.
Comment
Watch Question

Commented:
What size files are you dealing with? If under 50mb, just buffer it all.

>>Because it seems to take quite a bit
DoEvents on every loop will slow it down a lot and it's not required. You should only call it, at minimum, every 100ms. It should give enough time to display any form refreshes and interactions to process.

In this situation, you'll either have to sacrifice memory or speed. I'd rather sacrifice memory as most modern systems have quite a bit, and page-files are usually large.

Here's an example below (same concept as RichEdit). If file-sizes exceed 50mb, you'll probably wait a different approach.

Form1:
---------------
Option Explicit

Private m_fBuff() As String
Private Sub Form_Load()
    If (FileLoad("c:\test.txt") = True) Then
        Call MsgBox("Lines: " & CStr(UBound(m_fBuff) + 1), vbInformation)
    End If
End Sub
Private Function FileLoad(ByVal sFile As String) As Boolean
    Dim intFF As Integer
    Dim strBuff As String
    intFF = FreeFile
    If (Dir$(sFile, vbNormal Or vbHidden) <> vbNullString) Then
        Open sFile For Binary As intFF
            strBuff = Space$(LOF(intFF))
            Get #intFF, 1, strBuff
            m_fBuff = Split(strBuff, vbCrLf)
            FileLoad = True
        Close intFF
    End If
End Function

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts

Author

Commented:
Oh wow, I did not know that DoEvents slowed down the process so much!  I removed it just for testing, and it loaded it instantly!  That fixed my functions totally, I will add it to do it every 100 MS or so, thanks!
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.