Solved

Reading a text file backwards ?

Posted on 2002-07-22
23
489 Views
Last Modified: 2007-12-19
Hello i have a problem with readng a file.

Below is the format of the file i want to read

22/07/2002 11:22:07,Backup MISOutput file C:\MIS\Backup\MISOutput.bak
22/07/2002 11:23:08,Backup MISOutput file C:\MIS\Backup\MISOutput.bak
22/07/2002 11:23:09,Start Read from File C:\MIS\DATA\NTCv1000000314.txt
22/07/2002 11:27:15,Completed Read from File C:\MIS\DATA\NTCv1000000314.txt
23/07/2002 11:29:48,Backup MISOutput file C:\MIS\Backup\MISOutput.bak
23/07/2002 11:29:51,Start Read from File C:\MIS\DATA\NTCv1000000314.txt
23/07/2002 11:30:05,Completed Read from File C:\MIS\DATA\NTCv1000000314.txt
23/07/2002 11:30:41,Backup MISOutput file C:\MIS\Backup\MISOutput.bak
23/07/2002 11:30:41,Start Read from File C:\MIS\DATA\NTCv1000000314.txt

What i want to do is read this file and put the details into a list box. The file can get very big, so i only want to show a days worth at a time. The lastest information is at the end of the file.

So what i thought of doing was to read the file backwards, until the date was different and then display the information that i had read in the the list box.

I also wanted to add to buttons to the form, to allow the user to go back a day, or forward a day.

I have currently been using the FileSystemObject, but i am not sure it will allow me to read backwards through a file.

Can anyone help me achieve this, how do i do it ?

Thanks Very much



----

The code i have so far, just basically read the file and dumps it into a listbox

Set objTextLine = objFSO.OpenTextFile(strFilename, ForReading, False, TristateFalse)
     
    lblDate = "22/07/2002" ' Hard Coded
   
    While Not objTextLine.AtEndOfStream
        strReadLine = objTextLine.ReadLine
        lstLogDetails.AddItem (strReadLine)
    Wend
   
    Set objFSO = Nothing
    Set objTextLine = Nothing
0
Comment
Question by:Molko
  • 4
  • 4
  • 4
  • +6
23 Comments
 
LVL 16

Expert Comment

by:Richie_Simonetti
ID: 7169995
To me, yoiu have at least two choices:
1) Convert the file to a database. You could manage it kindly better.
2) Append text at beginning of the file, instead to the end.
0
 

Author Comment

by:Molko
ID: 7170007
It has to be a textfile and not a database.

Is appending to the beginning of the file easy, how would i do that.

SO are you saying that it is not possible to read the file backwards in VB. I know i C/C++, you have afile pointer which can be moved to anywhere in the file ???.

0
 
LVL 4

Expert Comment

by:gencross
ID: 7170013
Your best bet may be to do what you are doing, but put the file into an array or collection then add the elements to the listbox...

Collection example...

Dim colLine as Collection

Set colLine = New Collection

Set objTextLine = objFSO.OpenTextFile(strFilename, ForReading, False, TristateFalse)

   lblDate = "22/07/2002" ' Hard Coded
   
   While Not objTextLine.AtEndOfStream
       colLine.Add objTextLine.ReadLine
   Wend
   
   Set objFSO = Nothing
   Set objTextLine = Nothing

   'Read collection from end to beginning
   for x = colLine.Count to 1 Step -1
       lstLogDetails.AddItem (colLine(x))
   next

I may have forgotten something I am typing in the EE textbox, but if I did you still get the idea.
0
 

Author Comment

by:Molko
ID: 7170019
Gencross, i see what you are saying, but i think the file could get VERY big
0
 
LVL 4

Expert Comment

by:gencross
ID: 7170025
Since your file does not appear to be fixed length it is going to be very hard to read it from the end line by line.  File access in VB is very good (fast) it will be MUCH easier to just read the file from beginning then display it which ever way you want, just as my example above shows.
0
 
LVL 4

Expert Comment

by:gencross
ID: 7170030
If you are in control of writing the log file initially what about writing a new log file for each day?
0
 

Author Comment

by:Molko
ID: 7170045
I agree with reading the whole file all at once in principle, but i think the file could get to about 400,000+ records in length (i know it should be in a database, but its not). I am just reluctant to go about creating arrays/collections of that size.

Its not my decision, whether there is a logfile for each, BUT it is a very good point. I theory i could keep a rolling 30 days worth of info in 30 different file. But as i say its not my decision !
0
 
LVL 22

Expert Comment

by:rspahitz
ID: 7170048
Without FSO, you can handle this in one of two ways:

1) Open the file, move the filepointer to the len of the file, then start reading one character and decrement the filepointer until you find what you're seeking (like a CR/LF combo.)

2) Open the file, move the filepointer to the len of the file less some predetermined size, then read one block at a time and InstrRev-search for a CR/LF block and spluit that off as a separate line.

The first will be more direct but slower; the second will be faster but take more coding.  If you don't expect to have too many lines on a given day, the slower way is probably better since there is less chance for coding errors.
0
 

Author Comment

by:Molko
ID: 7170065
rspahitz, can you show me some example code. I dont know what you mean by filepointer in FSO.

Thanks
0
 
LVL 4

Expert Comment

by:gencross
ID: 7170067
That is pretty big to keep in memory.  If you cannot use a database or break up the log file daily another option could be to ask the user for criteria up front, such as a date.  You could then go through the file line by line searching for that date or even a date range and only return the lines for that date.
0
 
LVL 22

Expert Comment

by:rspahitz
ID: 7170076
Without FSO:

Private Sub Command1_Click()
  Const cFilename As String = "C:\logfile.txt"

  Dim strLine As String
  Dim lFilePosition As String
  Dim strChar As String * 1
 
  strLine = vbNullString
  Open cFilename For Binary As #1
  lFilePosition = LOF(1)
  Do
    Get #1, lFilePosition, strChar
    If strChar = vbLf Then
      Exit Do
    Else
      strLine = strChar & strLine
    End If
    lFilePosition = lFilePosition - 1
  Loop Until EOF(1)
  Close #1
  Text1.Text = strLine
End Sub

--
Assumptions:
1) You are only searching for the last line
2) The file does not end with a CR/LF

To search for more lines, simply change the "if" statement to match the desired criteria.
For example, every time you find a LF, assume the next char backwards is a CR and transfer the current "strLine" to an array.  If strLine does not contain today's date at the desired location, then exit the "do" loop otherwise clear strLine and continue.
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 16

Expert Comment

by:Richie_Simonetti
ID: 7170086
Even in the case that you cannot use a db...could you transform in in one just to read it.
I mean, it has some formatting inside that allows to create a db from it or openning like one with an ADO provider?
0
 
LVL 26

Expert Comment

by:EDDYKT
ID: 7170095
Take a look on this one which may relate to your question.
0
 
LVL 26

Expert Comment

by:EDDYKT
ID: 7170097
0
 
LVL 2

Expert Comment

by:vbPhil
ID: 7170261
Molko:

Grab a chunk of data at the end of the file, and examine it in an array.  You can tweak this example to fit your needs...

This example grabs a 32k chunk of data (which will fit nicely in memory, regardless of how large the file is).  If your data isn't found... grab another chunk of data 32k prior to where you grabbed the first chunk... etc.


Option Explicit

Private Sub Command1_Click()

  Dim fH As Integer
  Dim sFile As String
  Dim sBuff As String
  Dim lOffset As Long
  Dim lLength As Long
  Dim sArray() As String
  Dim lRecN As Long
   
 
  sFile = "c:\temp\somebigfile.txt"
 
  fH = FreeFile
  Open sFile For Binary As #fH
 
  lLength = LOF(fH)
  lOffset = lLength - (1024 * 32)
  If lOffset < 1 Then
    lOffset = 1
    sBuff = Space(lLength)
  Else
    sBuff = Space(1024 * 32)
  End If
   
  Get #fH, lOffset, sBuff
 
  Close #fH
 

 
 
  sArray = Split(sBuff, vbCrLf, -1, vbTextCompare)
 
  'results in zero based array
  'discard the first element, because it is likely not
  'on a record boundary.
 
  For lRecN = UBound(sArray) To 1 Step -1
    'examine the record for your date info here
  Next lRecN
 
  'if your data is not contained in this chunk, then start over, grabbing
  'a larger chunk of data.
   


End Sub




vbPhil

0
 
LVL 1

Expert Comment

by:superchook
ID: 7170757
You could work forward through the file - reading the date field for each line until you find the highest date.  FAST

(you could keep a byte count of where each date starts - this will make the next pass even faster than walking through the file from the start)

Then do the same thing again, ignoring all lines until you find the 'same' date as logged inthe first pass... also FAST (even faster if you jump straight in at the correct byte position)

Then copy out and process the records from the first matching line - to the end...  FAST


0
 
LVL 22

Expert Comment

by:rspahitz
ID: 7170768
Taking up from superchook's idea, if this log file will be static (i.e. rarely deleted) you can index each line's starting byte position and place that into a separate file.  Then you only have to parse through a small file to locate the start of each line.  And if done right, the index file can also contain the date the line was created so you only have to read the index file to figure out the start of any given line and all lines for a given date.
0
 
LVL 45

Expert Comment

by:aikimark
ID: 7170975
Molko,

You can use the Seek statement and function to quickly position your code in an open file.  You can still use a Line Input statement to read lines.  The quickest way to read the last day is to store the Seek() value from the previous day's display.

You can also use the Input() function to read as many bytes of data as you want.  You are then responsible for parsing this chunk of data.  Unfortunately, you can only read forward in a file with this format.

I'd say that superchook might have the fastest initial solution.  You will do well to save the date-start positions in another file to speed the processing for subsequent operations.
0
 
LVL 1

Expert Comment

by:Deverill
ID: 7171946
On a related topic, if you want the lines to appear in your listbox in reverse order you can use the code:

  lstLogDetails.AddItem (strReadLine),0

which will put each new line on line zero of the listbox (insert at the top).
0
 
LVL 45

Expert Comment

by:aikimark
ID: 7172100
But you would NEVER want to add 400000 rows to a listbox.

I recommend placing two controls on your form:
1. listbox into which you add one item for every day in your data file.
2. another control to show the records for that day.  You can use another listbox, multi-line textbox with scrollbars, two column grid control (one for time and the other for event text).  The textbox will probably be the fastest load.

When the user selects a date, clear the other control a fill it with the data from that date.

Your UI now allows the user to directly display any date without having to look at intervening days' data.
0
 
LVL 2

Accepted Solution

by:
vbPhil earned 100 total points
ID: 7172194
I still believe that you are best off, doing something along the lines that I described earlier...

Grab a hunk of data...  examine it for your dates...

Perhaps this makes mores sense:

1.Get the length of the file...
2.Calculate the approximate midpoint and grab a hunk of data...
3.Find the first occurance of CRLF
  lStart = instr(1,sReadBuff,vbcrlf,vbTextCompare)+2

You now have the beginning of the first complete record of the hunk of data.

Check the date...  

If mid(sReadBuff,lStart,10) > sDateToFind Then
  'You haven't gone back far enough
  'so grab a hunk of data further up
else
  If mid(sReadBuff,lStart,10) < sDateToFind Then
    'You are to far up in the file
    'Grab a hunk of data further down
  else
    'You are in the dates you are looking for
    'grab a hunk of data a little further up
    'to find the beginning of the date range.
  End If
End If

Once you are close... use the Seek Statement

Seek [#]filenumber, position to give it a starting
point where you can then use Line Input # to pull
sequential records.

You can also just use the Seek statement to hop around
within the file boundaries.  Discard the first Line Input # results, as it will probably be incomplete, but the next Line Input read should give you a valid record.

Note:  If you use the hunk grabbing technique... you can just scan the entire hunk for the date you are looking for.

For example... lets say you grab a 16k chunk of the file.

If instr(1,sReadbuffer,sDateToFind,vbTextCompare) > 0 then
  'you know you are in the right neigborhood.
End If




0
 
LVL 22

Expert Comment

by:rspahitz
ID: 7172247
Molko, there are several good solutions here, depending on how fast you want it, how simple you want it, and how much information you need from the file.  Any thoughts?

Also, you asked, "Is appending to the beginning of the file easy"

Yes.  Open a new file and insert "today's" log file, then concatenate the previous log file by either opening it, reading the contents and writing to the new file (then closing both) or by simply using a simple old DOS command:  COPY file1+file2 file3

Sample code to "pre-pend" a line:

dim strLogFile as string
dim strCommand as string
dim strOriginalFile as string

strLogFile = "C:\SomePath\logfilename.log"

' Create the new log line
open "C:\logline.txt" for output as #1
print #1, strLogInfo
close #1

' Determine whether you're running 95/98/ME or NT/2000/XP
strCommand = environ("ComSpec")

' Concatenate the files (new one at the top of the log file
shell(strCommand & " /c COPY C:\logline+" & strOriginalFile & " " & strOriginalFile)

' Remove the temp file
kill strLogInfo
0
 
LVL 2

Expert Comment

by:vbPhil
ID: 7175310
Code to index file (using your sample data), displaying each unique date in a list box.  Click on list box, and it will go out and read that line.  You can modify it to read an entire day.

This will run very quickly, as you are not displaying records as they are read.  You could store the index in a file for later retrieval if desired.  Use the UDT shown to record the data from the listbox into a file.


Option Explicit

'For example...
'Create Form1
'Place Command1 and List1 on it
'Paste this code into the form module


Private Type DateLog
  logDate  As Date
  logEntry As String
  offset   As Long
End Type
Private mDateIdx As DateLog


Private Sub Command1_Click()

  Dim fH As Integer
  Dim sFile As String
  Dim sBuff As String
  Dim dteDate As Date
  Dim dtePrevDate As Date
  Dim lRecN As Long
 
  sFile = Trim(App.Path) & "\datelog.txt"
  fH = FreeFile
  Open sFile For Input As #fH
 
  dtePrevDate = CDate("01/01/1800")
 
  Do
   
    If EOF(fH) = True Then Exit Do
   
    Line Input #fH, sBuff
   
    dteDate = CDate(Mid(sBuff, 1, 10))
   
    If dteDate <> dtePrevDate Then
     
      dtePrevDate = dteDate
     
      mDateIdx.logDate = dteDate
      mDateIdx.logEntry = sBuff
      mDateIdx.offset = Seek(fH) - (Len(sBuff) + 2)
     
      List1.AddItem Format(mDateIdx.logDate, "yyyy/mm/dd")
      List1.ItemData(List1.NewIndex) = mDateIdx.offset
     
      'Just clears entry for next pass
      mDateIdx.logDate = 0
      mDateIdx.logEntry = ""
      mDateIdx.offset = 0
     
    End If
   
  Loop
 
  Close #fH
 
 
 

End Sub

Private Sub List1_Click()

  Dim fH As Integer
  Dim sFile As String
  Dim sBuff As String
  Dim lOffset As Long
 
 
  If List1.ListIndex >= 0 Then
    lOffset = List1.ItemData(List1.ListIndex)
  Else
    Exit Sub
  End If
 
  sFile = Trim(App.Path) & "\datelog.txt"
  fH = FreeFile
  Open sFile For Input As #fH
 
  Seek #fH, lOffset

  Line Input #fH, sBuff
 
  Close #fH
 
  MsgBox sBuff
 


End Sub
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
VBA: Add rows to listbox based on criteria 7 64
Access 2013 combo box not working 3 25
Copy a row 12 52
Spell Check in VB6 13 42
Introduction In a recent article (http://www.experts-exchange.com/A_7811-A-Better-Concatenate-Function.html) for the Excel community, I showed an improved version of the Excel Concatenate() function.  While writing that article I realized that no o…
If you have ever used Microsoft Word then you know that it has a good spell checker and it may have occurred to you that the ability to check spelling might be a nice piece of functionality to add to certain applications of yours. Well the code that…
As developers, we are not limited to the functions provided by the VBA language. In addition, we can call the functions that are part of the Windows operating system. These functions are part of the Windows API (Application Programming Interface). U…
Get people started with the process of using Access VBA to control Excel using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Excel. Using automation, an Access application can laun…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now