Reading text files by line number

Hi icredes,

Are the lines in your text file fixed width or variable length?

Define "large files" for us.  Approx how many lines are we talking about here?

The lines are variantly fixed (if that helps)
   There are some lines that X number and other lines are Y and other lines are Z length, but nothing consistant enough to calculate.

The file size would get up to several megabytes.  This application is to process transactions coming in via an external dialer, the size of the file is unknown, but 1,000,000 transactions would make the file about 60MB

The current activity does not get anywhere near this size, but great marketing is being put forth and activity will increase greatly.

I am more concerned about degrading performance over time.  

There are some lines that will be ignored depending on how they are interpereted.

--------------------------------------------------------------
I hope this answers your question

you can read the file all into 1 var and then split it into an array, and then you could loop through the array backwards, for example...

Open FileName For Input As #1
ReadFile = Input(LOF(1), 1)                         ' read whole file into variable called Readfile
Close #1

LGP() = Split(ReadFile(FileName), vbCrLf)   ' split lines array into seperate lines using line feed as delimiter


for i = ubound(lgp) to 0 step -1

next i

How about this approach...

We store the file length in the registry.  When we go to read the file we first check the registry for the previous file length.  (If there is none then we use a length of 1 which starts reading at the beginning of the file.)  Then we use that previous file length to set the file position using the Seek() function.  This allows us to skip over the data we previously read and processed.  Next we calculate how much file is left by subtracting the file position from the file length.  Now we read in the remaining unread portion of the file into a string and split it in a similar fashion that giff87 has described.  Process the lines of the file.  Finally, put the current file length back into the registry so that next time we read we skip over what we just processed...

Option Explicit

Private fileName As String

Private Sub Form_Load()
    fileName = "c:\someFile.txt"
End Sub

Private Sub Command1_Click()
    Dim ff As Integer
    Dim filePos As Long
    Dim remainingSize As Long
    Dim remainingFile As String
    Dim fileLength As Long
    Dim lines As Variant
    Dim i As Long
   
    ' make sure the file exists
    If Dir(fileName) <> "" Then
   
        ' get the last read file position from the registry
        filePos = GetSetting("MyApp", "Settings", "filePos", "1")
        Debug.Print "filePos = " & filePos
       
        ' open the file
        ff = FreeFile
        Open fileName For Binary Access Read As #ff
       
        ' get the files current length
        fileLength = LOF(ff)
        Debug.Print "fileLength = " & fileLength
       
        ' make sure the file position makes sense with respect to the file size
        If fileLength > filePos Then
            ' skip over the parts we have already read
            Seek ff, filePos
           
            ' calculate how much of the file is left to be read
            remainingSize = fileLength - filePos + 1
           
            ' read in the rest of the file
            remainingFile = Input(remainingSize, ff)
           
            ' do something with remainingFile
            lines = Split(remainingFile, vbCrLf)
            For i = 0 To UBound(lines)
                Debug.Print i & ": " & lines(i)
            Next i
           
            ' save the file position so next time we skip over what we just read
            SaveSetting "MyApp", "Settings", "filePos", fileLength
           
        ElseIf fileLength = filePos Then
            MsgBox "No new lines to read.", vbInformation, "File size same as last time"
           
        Else
            MsgBox "Last file position is greater than current file size.", vbCritical, "File Changed/Modified"
           
        End If
       
        ' close the file
        Close #ff
    Else
        MsgBox fileName, vbCritical, "File Not Found"
    End If
End Sub

Private Sub Command2_Click()
    ' reset the file position in the registry
    ' this will allow the entire file to be read again
    SaveSetting "MyApp", "Settings", "filePos", "1"
End Sub

would saving the entire file eat up memory resources if I save a file that is 10MB or higher?

Explain in a bit more detail.  I'm not sure what you are asking...

Dim FSO As Object, T As Object
Set FSO = CreateObject("Scripting.FileSystemObject")
Set T = FSO.OpenTextFile("D:\test.txt", 1, False, 0) 'File to open and read line number
Do While Not T.AtEndOfStream
lineread = T.ReadLine
If T.Line = 4 Then 'Line number you want to read
MsgBox T.ReadLine
End If
Loop
Set FSO = Nothing
Set T = Nothing

Say I have a file that has 20000 lines
and all I really need to do is read the last 15 lines

I am looking for a faster way to get read the lines without reading the entire file
so I would like to either
start reading at line

19985 and finish out the file

or

start at 20000  and read the last 15 lines down

I like that idea

let me work on it and see how it works

thank you so much for your help Idle Mind

Instead of using a text file to store quite so many records, perhaps it would be more efficient to use a database? You can read records from a database at will.