Solved

Reading UNIX format (LF delimited) text files - BUG

Posted on 2002-05-17
6
182 Views
Last Modified: 2010-04-04
The driver for reading UNIX files in my previously asked question (qid=20295121) apparently has a little bug in it.  On certain files it will get to the end, then rather than properly detecting EOF, a ReadLn will forever keep returning blank lines.

I suspect it has to do with the last line coming out to the perfect length in the read buffer or something.  Below is the code: can anyone spot the problem?   I have two short sample files (1 good 1 bad) ready for testing.

=======================================================
unit StreamFile;
{
Use AssignStreamFile instead of AssignFile, and you can
read either CRLF *or* LF delimited files.


        Unix Stream File Interface
        Copyright 1996 by John Miano Software
        miano@worldnet.att.net

        5/1/2002 C. Boling  Status vars not used; removed references to prevent compiler warnings.
                            Also, remove any CR's on Read.
}
interface

Uses
  SysUtils ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;

implementation

Const
  BufferSize = 128 ;

Type
  TStreamBuffer = Array [1..High (Integer)] of Char ;
  TStreamBufferPointer = ^TStreamBuffer ;

  TStreamFileRecord = Record
    Case Integer Of
      1:
        (
          Filehandle : Integer ;
          Buffer : TStreamBufferPointer ;
          BufferOffset : Integer ;
          ReadCount : Integer ;
        ) ;
      2:
        (
          Dummy : Array [1 .. 32] Of Char
        )
    End ;


Function StreamFileOpen (var F : TTextRec) : Integer ;
  //Var
    //Status : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    GetMem (Buffer, BufferSize) ;
    Case F.Mode Of
      fmInput:
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone) ;
      fmOutput:
        FileHandle := FileCreate (StrPas (F.Name)) ;
      fmInOut:
        Begin
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone Or
fmOpenWrite or fmOpenRead) ;
        If FileHandle <> -1 Then
          //status := FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
          FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
        F.Mode := fmOutput ;
        End ;
      End ;
    BufferOffset := 0 ;
    ReadCount := 0 ;
    F.BufEnd := 0 ;  { If this is not here it thinks we are at eof. }
    If FileHandle = -1 Then
      Result := -1
    Else
      Result := 0 ;
    End ;
  End ;

Function StreamFileInOut (var F : TTextRec) : Integer ;
  Procedure Read (var Data : TStreamFileRecord) ;
    Procedure CopyData ;
      Begin
      While (F.BufEnd < Sizeof (F.Buffer) - 2)
            And (Data.BufferOffset <= Data.ReadCount)
            And (Data.Buffer [Data.BufferOffset] <> #10) Do
        Begin
        {start 5-1-2002}
        //F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
        //Inc (Data.BufferOffset) ;
        //Inc (F.BufEnd) ;
        if Data.Buffer [Data.BufferOffset] <> #13 then
        begin
          F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
          Inc (F.BufEnd) ;
        end;
        Inc (Data.BufferOffset) ;
        {end 5-1-2002}
        End ;
      If Data.Buffer [Data.BufferOffset] = #10 Then
        Begin
        F.Buffer [F.BufEnd] := #13 ;
        Inc (F.BufEnd) ;
        F.Buffer [F.BufEnd] := #10 ;
        Inc (F.BufEnd) ;
        Inc (Data.BufferOffset) ;
        End ;
      End ;

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;

  Procedure Write (var Data : TStreamFileRecord) ;
    Var
      //Status : Integer ;
      Destination : Integer ;
      II : Integer ;
    Begin
    With TStreamFileRecord (F.UserData) Do
      Begin
      Destination := 0 ;
      For II := 0 To F.BufPos - 1 Do
        Begin
        If F.Buffer [II] <> #13 Then
          Begin
          Inc (Destination) ;
          Buffer^[Destination] := F.Buffer [II] ;
          End ;
        End ;
      //Status := FileWrite (FileHandle, Buffer^, Destination) ;
      FileWrite (FileHandle, Buffer^, Destination) ;
      F.BufPos := 0 ;
      Result := 0 ;
      End ;
    End ;
  Begin
  Case F.Mode Of
    fmInput:
      Read (TStreamFileRecord (F.UserData)) ;
    fmOutput:
      Write (TStreamFileRecord (F.UserData)) ;
    End ;
  End ;

Function StreamFileFlush (var F : TTextRec) : Integer ;
  Begin
  Result := 0 ;
  End ;

Function StreamFileClose (var F : TTextRec) : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    FreeMem (Buffer) ;
    FileClose (FileHandle) ;
    End ;
  Result := 0 ;
  End ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;
  Begin
  With TTextRec (F) Do
    Begin
    Mode := fmClosed ;
    BufPtr := @Buffer ;
    BufSize := Sizeof (Buffer) ;
    OpenFunc := @StreamFileOpen ;
    InOutFunc := @StreamFileInOut ;
    FlushFunc := @StreamFileFlush ;
    CloseFunc := @StreamFileClose ;
    StrPLCopy (Name, FileName, Sizeof(Name) - 1) ;
    End ;
  End ;
end.

0
Comment
Question by:charles_ebs
6 Comments
 
LVL 33

Expert Comment

by:Slick812
ID: 7017127
seems like an easier way would be to load a FileStream into a String variable and then use the AdjustLineBreaks( ) function on that string
0
 

Author Comment

by:charles_ebs
ID: 7017136
...unless you have a multi-gigabyte file and don't have that much RAM. :-)
0
 
LVL 14

Expert Comment

by:DragonSlayer
ID: 7017805
You should consider loading your text file into a TStringList, it will detect Unix/Windows linebreaks correctly (again, as what charles said, you shouldn't do that if you have a HUGE file).

You can then read it back line by line using the Strings[] property of the StringList.


HTH
DragonSlayer
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 9

Expert Comment

by:ITugay
ID: 7018052
Hi charles_ebs,

be carefull with files bigger then 2GB.

-------
Igor.
0
 

Author Comment

by:charles_ebs
ID: 7024898
Found the bug.

The routine reads the file in 128-byte chunks.   The conditions under which the bug occured was if the 1st byte read into the buffer during the last read [that returned data] from the file happened to be a LF character.  If so, then it would keep returning phantom blank lines forever.

Here's the patched code section, with a comment marking the line that I added (look for the string "BUGFIX"):

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      If (Data.ReadCount > 0) then {BUGFIX C. Boling 5/21/01.  Added this conditional; otherwise, if 1st char in buffer happens to be #10 at eof, it would keep returning blank lines indefinitely.}
        CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;
 
0
 
LVL 6

Accepted Solution

by:
Mindphaser earned 0 total points
ID: 7025256
Charles

Thanks for the solution!

I will refund your points and move this to PAQ.

** Mindphaser - Community Support Moderator **
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now