Solved

Reading UNIX format (LF delimited) text files - BUG

Posted on 2002-05-17
6
187 Views
Last Modified: 2010-04-04
The driver for reading UNIX files in my previously asked question (qid=20295121) apparently has a little bug in it.  On certain files it will get to the end, then rather than properly detecting EOF, a ReadLn will forever keep returning blank lines.

I suspect it has to do with the last line coming out to the perfect length in the read buffer or something.  Below is the code: can anyone spot the problem?   I have two short sample files (1 good 1 bad) ready for testing.

=======================================================
unit StreamFile;
{
Use AssignStreamFile instead of AssignFile, and you can
read either CRLF *or* LF delimited files.


        Unix Stream File Interface
        Copyright 1996 by John Miano Software
        miano@worldnet.att.net

        5/1/2002 C. Boling  Status vars not used; removed references to prevent compiler warnings.
                            Also, remove any CR's on Read.
}
interface

Uses
  SysUtils ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;

implementation

Const
  BufferSize = 128 ;

Type
  TStreamBuffer = Array [1..High (Integer)] of Char ;
  TStreamBufferPointer = ^TStreamBuffer ;

  TStreamFileRecord = Record
    Case Integer Of
      1:
        (
          Filehandle : Integer ;
          Buffer : TStreamBufferPointer ;
          BufferOffset : Integer ;
          ReadCount : Integer ;
        ) ;
      2:
        (
          Dummy : Array [1 .. 32] Of Char
        )
    End ;


Function StreamFileOpen (var F : TTextRec) : Integer ;
  //Var
    //Status : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    GetMem (Buffer, BufferSize) ;
    Case F.Mode Of
      fmInput:
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone) ;
      fmOutput:
        FileHandle := FileCreate (StrPas (F.Name)) ;
      fmInOut:
        Begin
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone Or
fmOpenWrite or fmOpenRead) ;
        If FileHandle <> -1 Then
          //status := FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
          FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
        F.Mode := fmOutput ;
        End ;
      End ;
    BufferOffset := 0 ;
    ReadCount := 0 ;
    F.BufEnd := 0 ;  { If this is not here it thinks we are at eof. }
    If FileHandle = -1 Then
      Result := -1
    Else
      Result := 0 ;
    End ;
  End ;

Function StreamFileInOut (var F : TTextRec) : Integer ;
  Procedure Read (var Data : TStreamFileRecord) ;
    Procedure CopyData ;
      Begin
      While (F.BufEnd < Sizeof (F.Buffer) - 2)
            And (Data.BufferOffset <= Data.ReadCount)
            And (Data.Buffer [Data.BufferOffset] <> #10) Do
        Begin
        {start 5-1-2002}
        //F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
        //Inc (Data.BufferOffset) ;
        //Inc (F.BufEnd) ;
        if Data.Buffer [Data.BufferOffset] <> #13 then
        begin
          F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
          Inc (F.BufEnd) ;
        end;
        Inc (Data.BufferOffset) ;
        {end 5-1-2002}
        End ;
      If Data.Buffer [Data.BufferOffset] = #10 Then
        Begin
        F.Buffer [F.BufEnd] := #13 ;
        Inc (F.BufEnd) ;
        F.Buffer [F.BufEnd] := #10 ;
        Inc (F.BufEnd) ;
        Inc (Data.BufferOffset) ;
        End ;
      End ;

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;

  Procedure Write (var Data : TStreamFileRecord) ;
    Var
      //Status : Integer ;
      Destination : Integer ;
      II : Integer ;
    Begin
    With TStreamFileRecord (F.UserData) Do
      Begin
      Destination := 0 ;
      For II := 0 To F.BufPos - 1 Do
        Begin
        If F.Buffer [II] <> #13 Then
          Begin
          Inc (Destination) ;
          Buffer^[Destination] := F.Buffer [II] ;
          End ;
        End ;
      //Status := FileWrite (FileHandle, Buffer^, Destination) ;
      FileWrite (FileHandle, Buffer^, Destination) ;
      F.BufPos := 0 ;
      Result := 0 ;
      End ;
    End ;
  Begin
  Case F.Mode Of
    fmInput:
      Read (TStreamFileRecord (F.UserData)) ;
    fmOutput:
      Write (TStreamFileRecord (F.UserData)) ;
    End ;
  End ;

Function StreamFileFlush (var F : TTextRec) : Integer ;
  Begin
  Result := 0 ;
  End ;

Function StreamFileClose (var F : TTextRec) : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    FreeMem (Buffer) ;
    FileClose (FileHandle) ;
    End ;
  Result := 0 ;
  End ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;
  Begin
  With TTextRec (F) Do
    Begin
    Mode := fmClosed ;
    BufPtr := @Buffer ;
    BufSize := Sizeof (Buffer) ;
    OpenFunc := @StreamFileOpen ;
    InOutFunc := @StreamFileInOut ;
    FlushFunc := @StreamFileFlush ;
    CloseFunc := @StreamFileClose ;
    StrPLCopy (Name, FileName, Sizeof(Name) - 1) ;
    End ;
  End ;
end.

0
Comment
Question by:charles_ebs
6 Comments
 
LVL 34

Expert Comment

by:Slick812
ID: 7017127
seems like an easier way would be to load a FileStream into a String variable and then use the AdjustLineBreaks( ) function on that string
0
 

Author Comment

by:charles_ebs
ID: 7017136
...unless you have a multi-gigabyte file and don't have that much RAM. :-)
0
 
LVL 14

Expert Comment

by:DragonSlayer
ID: 7017805
You should consider loading your text file into a TStringList, it will detect Unix/Windows linebreaks correctly (again, as what charles said, you shouldn't do that if you have a HUGE file).

You can then read it back line by line using the Strings[] property of the StringList.


HTH
DragonSlayer
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 9

Expert Comment

by:ITugay
ID: 7018052
Hi charles_ebs,

be carefull with files bigger then 2GB.

-------
Igor.
0
 

Author Comment

by:charles_ebs
ID: 7024898
Found the bug.

The routine reads the file in 128-byte chunks.   The conditions under which the bug occured was if the 1st byte read into the buffer during the last read [that returned data] from the file happened to be a LF character.  If so, then it would keep returning phantom blank lines forever.

Here's the patched code section, with a comment marking the line that I added (look for the string "BUGFIX"):

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      If (Data.ReadCount > 0) then {BUGFIX C. Boling 5/21/01.  Added this conditional; otherwise, if 1st char in buffer happens to be #10 at eof, it would keep returning blank lines indefinitely.}
        CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;
 
0
 
LVL 6

Accepted Solution

by:
Mindphaser earned 0 total points
ID: 7025256
Charles

Thanks for the solution!

I will refund your points and move this to PAQ.

** Mindphaser - Community Support Moderator **
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
error 1.1 400 Bad request idhttp delphi 18 199
Intraweb download file link ? 1 167
When i run adoquery my application freezes 26 188
Delphi Firemonkey send email on Android 1 38
This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
Objective: - This article will help user in how to convert their numeric value become words. How to use 1. You can copy this code in your Unit as function 2. than you can perform your function by type this code The Code   (CODE) The Im…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the admini…

713 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question