Solved

Reading UNIX format (LF delimited) text files - BUG

Posted on 2002-05-17
6
183 Views
Last Modified: 2010-04-04
The driver for reading UNIX files in my previously asked question (qid=20295121) apparently has a little bug in it.  On certain files it will get to the end, then rather than properly detecting EOF, a ReadLn will forever keep returning blank lines.

I suspect it has to do with the last line coming out to the perfect length in the read buffer or something.  Below is the code: can anyone spot the problem?   I have two short sample files (1 good 1 bad) ready for testing.

=======================================================
unit StreamFile;
{
Use AssignStreamFile instead of AssignFile, and you can
read either CRLF *or* LF delimited files.


        Unix Stream File Interface
        Copyright 1996 by John Miano Software
        miano@worldnet.att.net

        5/1/2002 C. Boling  Status vars not used; removed references to prevent compiler warnings.
                            Also, remove any CR's on Read.
}
interface

Uses
  SysUtils ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;

implementation

Const
  BufferSize = 128 ;

Type
  TStreamBuffer = Array [1..High (Integer)] of Char ;
  TStreamBufferPointer = ^TStreamBuffer ;

  TStreamFileRecord = Record
    Case Integer Of
      1:
        (
          Filehandle : Integer ;
          Buffer : TStreamBufferPointer ;
          BufferOffset : Integer ;
          ReadCount : Integer ;
        ) ;
      2:
        (
          Dummy : Array [1 .. 32] Of Char
        )
    End ;


Function StreamFileOpen (var F : TTextRec) : Integer ;
  //Var
    //Status : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    GetMem (Buffer, BufferSize) ;
    Case F.Mode Of
      fmInput:
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone) ;
      fmOutput:
        FileHandle := FileCreate (StrPas (F.Name)) ;
      fmInOut:
        Begin
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone Or
fmOpenWrite or fmOpenRead) ;
        If FileHandle <> -1 Then
          //status := FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
          FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
        F.Mode := fmOutput ;
        End ;
      End ;
    BufferOffset := 0 ;
    ReadCount := 0 ;
    F.BufEnd := 0 ;  { If this is not here it thinks we are at eof. }
    If FileHandle = -1 Then
      Result := -1
    Else
      Result := 0 ;
    End ;
  End ;

Function StreamFileInOut (var F : TTextRec) : Integer ;
  Procedure Read (var Data : TStreamFileRecord) ;
    Procedure CopyData ;
      Begin
      While (F.BufEnd < Sizeof (F.Buffer) - 2)
            And (Data.BufferOffset <= Data.ReadCount)
            And (Data.Buffer [Data.BufferOffset] <> #10) Do
        Begin
        {start 5-1-2002}
        //F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
        //Inc (Data.BufferOffset) ;
        //Inc (F.BufEnd) ;
        if Data.Buffer [Data.BufferOffset] <> #13 then
        begin
          F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
          Inc (F.BufEnd) ;
        end;
        Inc (Data.BufferOffset) ;
        {end 5-1-2002}
        End ;
      If Data.Buffer [Data.BufferOffset] = #10 Then
        Begin
        F.Buffer [F.BufEnd] := #13 ;
        Inc (F.BufEnd) ;
        F.Buffer [F.BufEnd] := #10 ;
        Inc (F.BufEnd) ;
        Inc (Data.BufferOffset) ;
        End ;
      End ;

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;

  Procedure Write (var Data : TStreamFileRecord) ;
    Var
      //Status : Integer ;
      Destination : Integer ;
      II : Integer ;
    Begin
    With TStreamFileRecord (F.UserData) Do
      Begin
      Destination := 0 ;
      For II := 0 To F.BufPos - 1 Do
        Begin
        If F.Buffer [II] <> #13 Then
          Begin
          Inc (Destination) ;
          Buffer^[Destination] := F.Buffer [II] ;
          End ;
        End ;
      //Status := FileWrite (FileHandle, Buffer^, Destination) ;
      FileWrite (FileHandle, Buffer^, Destination) ;
      F.BufPos := 0 ;
      Result := 0 ;
      End ;
    End ;
  Begin
  Case F.Mode Of
    fmInput:
      Read (TStreamFileRecord (F.UserData)) ;
    fmOutput:
      Write (TStreamFileRecord (F.UserData)) ;
    End ;
  End ;

Function StreamFileFlush (var F : TTextRec) : Integer ;
  Begin
  Result := 0 ;
  End ;

Function StreamFileClose (var F : TTextRec) : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    FreeMem (Buffer) ;
    FileClose (FileHandle) ;
    End ;
  Result := 0 ;
  End ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;
  Begin
  With TTextRec (F) Do
    Begin
    Mode := fmClosed ;
    BufPtr := @Buffer ;
    BufSize := Sizeof (Buffer) ;
    OpenFunc := @StreamFileOpen ;
    InOutFunc := @StreamFileInOut ;
    FlushFunc := @StreamFileFlush ;
    CloseFunc := @StreamFileClose ;
    StrPLCopy (Name, FileName, Sizeof(Name) - 1) ;
    End ;
  End ;
end.

0
Comment
Question by:charles_ebs
6 Comments
 
LVL 33

Expert Comment

by:Slick812
ID: 7017127
seems like an easier way would be to load a FileStream into a String variable and then use the AdjustLineBreaks( ) function on that string
0
 

Author Comment

by:charles_ebs
ID: 7017136
...unless you have a multi-gigabyte file and don't have that much RAM. :-)
0
 
LVL 14

Expert Comment

by:DragonSlayer
ID: 7017805
You should consider loading your text file into a TStringList, it will detect Unix/Windows linebreaks correctly (again, as what charles said, you shouldn't do that if you have a HUGE file).

You can then read it back line by line using the Strings[] property of the StringList.


HTH
DragonSlayer
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 9

Expert Comment

by:ITugay
ID: 7018052
Hi charles_ebs,

be carefull with files bigger then 2GB.

-------
Igor.
0
 

Author Comment

by:charles_ebs
ID: 7024898
Found the bug.

The routine reads the file in 128-byte chunks.   The conditions under which the bug occured was if the 1st byte read into the buffer during the last read [that returned data] from the file happened to be a LF character.  If so, then it would keep returning phantom blank lines forever.

Here's the patched code section, with a comment marking the line that I added (look for the string "BUGFIX"):

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      If (Data.ReadCount > 0) then {BUGFIX C. Boling 5/21/01.  Added this conditional; otherwise, if 1st char in buffer happens to be #10 at eof, it would keep returning blank lines indefinitely.}
        CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;
 
0
 
LVL 6

Accepted Solution

by:
Mindphaser earned 0 total points
ID: 7025256
Charles

Thanks for the solution!

I will refund your points and move this to PAQ.

** Mindphaser - Community Support Moderator **
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
calling procedure inside activexForm from secondform 1 232
How to use Hashing sha1 in Delphi 2010 4 206
creating threads in delphi 1 84
Create a path if not exists 7 69
Hello everybody This Article will show you how to validate number with TEdit control, What's the TEdit control? TEdit is a standard Windows edit control on a form, it allows to user to write, read and copy/paste single line of text. Usua…
Introduction Raise your hands if you were as upset with FireMonkey as I was when I discovered that there was no TListview.  I use TListView in almost all of my applications I've written, and I was not going to compromise by resorting to TStringGrid…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
Both in life and business – not all partnerships are created equal. As the demand for cloud services increases, so do the number of self-proclaimed cloud partners. Asking the right questions up front in the partnership, will enable both parties …

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now