Solved

Reading UNIX format (LF delimited) text files - BUG

Posted on 2002-05-17
6
184 Views
Last Modified: 2010-04-04
The driver for reading UNIX files in my previously asked question (qid=20295121) apparently has a little bug in it.  On certain files it will get to the end, then rather than properly detecting EOF, a ReadLn will forever keep returning blank lines.

I suspect it has to do with the last line coming out to the perfect length in the read buffer or something.  Below is the code: can anyone spot the problem?   I have two short sample files (1 good 1 bad) ready for testing.

=======================================================
unit StreamFile;
{
Use AssignStreamFile instead of AssignFile, and you can
read either CRLF *or* LF delimited files.


        Unix Stream File Interface
        Copyright 1996 by John Miano Software
        miano@worldnet.att.net

        5/1/2002 C. Boling  Status vars not used; removed references to prevent compiler warnings.
                            Also, remove any CR's on Read.
}
interface

Uses
  SysUtils ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;

implementation

Const
  BufferSize = 128 ;

Type
  TStreamBuffer = Array [1..High (Integer)] of Char ;
  TStreamBufferPointer = ^TStreamBuffer ;

  TStreamFileRecord = Record
    Case Integer Of
      1:
        (
          Filehandle : Integer ;
          Buffer : TStreamBufferPointer ;
          BufferOffset : Integer ;
          ReadCount : Integer ;
        ) ;
      2:
        (
          Dummy : Array [1 .. 32] Of Char
        )
    End ;


Function StreamFileOpen (var F : TTextRec) : Integer ;
  //Var
    //Status : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    GetMem (Buffer, BufferSize) ;
    Case F.Mode Of
      fmInput:
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone) ;
      fmOutput:
        FileHandle := FileCreate (StrPas (F.Name)) ;
      fmInOut:
        Begin
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone Or
fmOpenWrite or fmOpenRead) ;
        If FileHandle <> -1 Then
          //status := FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
          FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
        F.Mode := fmOutput ;
        End ;
      End ;
    BufferOffset := 0 ;
    ReadCount := 0 ;
    F.BufEnd := 0 ;  { If this is not here it thinks we are at eof. }
    If FileHandle = -1 Then
      Result := -1
    Else
      Result := 0 ;
    End ;
  End ;

Function StreamFileInOut (var F : TTextRec) : Integer ;
  Procedure Read (var Data : TStreamFileRecord) ;
    Procedure CopyData ;
      Begin
      While (F.BufEnd < Sizeof (F.Buffer) - 2)
            And (Data.BufferOffset <= Data.ReadCount)
            And (Data.Buffer [Data.BufferOffset] <> #10) Do
        Begin
        {start 5-1-2002}
        //F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
        //Inc (Data.BufferOffset) ;
        //Inc (F.BufEnd) ;
        if Data.Buffer [Data.BufferOffset] <> #13 then
        begin
          F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
          Inc (F.BufEnd) ;
        end;
        Inc (Data.BufferOffset) ;
        {end 5-1-2002}
        End ;
      If Data.Buffer [Data.BufferOffset] = #10 Then
        Begin
        F.Buffer [F.BufEnd] := #13 ;
        Inc (F.BufEnd) ;
        F.Buffer [F.BufEnd] := #10 ;
        Inc (F.BufEnd) ;
        Inc (Data.BufferOffset) ;
        End ;
      End ;

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;

  Procedure Write (var Data : TStreamFileRecord) ;
    Var
      //Status : Integer ;
      Destination : Integer ;
      II : Integer ;
    Begin
    With TStreamFileRecord (F.UserData) Do
      Begin
      Destination := 0 ;
      For II := 0 To F.BufPos - 1 Do
        Begin
        If F.Buffer [II] <> #13 Then
          Begin
          Inc (Destination) ;
          Buffer^[Destination] := F.Buffer [II] ;
          End ;
        End ;
      //Status := FileWrite (FileHandle, Buffer^, Destination) ;
      FileWrite (FileHandle, Buffer^, Destination) ;
      F.BufPos := 0 ;
      Result := 0 ;
      End ;
    End ;
  Begin
  Case F.Mode Of
    fmInput:
      Read (TStreamFileRecord (F.UserData)) ;
    fmOutput:
      Write (TStreamFileRecord (F.UserData)) ;
    End ;
  End ;

Function StreamFileFlush (var F : TTextRec) : Integer ;
  Begin
  Result := 0 ;
  End ;

Function StreamFileClose (var F : TTextRec) : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    FreeMem (Buffer) ;
    FileClose (FileHandle) ;
    End ;
  Result := 0 ;
  End ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;
  Begin
  With TTextRec (F) Do
    Begin
    Mode := fmClosed ;
    BufPtr := @Buffer ;
    BufSize := Sizeof (Buffer) ;
    OpenFunc := @StreamFileOpen ;
    InOutFunc := @StreamFileInOut ;
    FlushFunc := @StreamFileFlush ;
    CloseFunc := @StreamFileClose ;
    StrPLCopy (Name, FileName, Sizeof(Name) - 1) ;
    End ;
  End ;
end.

0
Comment
Question by:charles_ebs
6 Comments
 
LVL 33

Expert Comment

by:Slick812
ID: 7017127
seems like an easier way would be to load a FileStream into a String variable and then use the AdjustLineBreaks( ) function on that string
0
 

Author Comment

by:charles_ebs
ID: 7017136
...unless you have a multi-gigabyte file and don't have that much RAM. :-)
0
 
LVL 14

Expert Comment

by:DragonSlayer
ID: 7017805
You should consider loading your text file into a TStringList, it will detect Unix/Windows linebreaks correctly (again, as what charles said, you shouldn't do that if you have a HUGE file).

You can then read it back line by line using the Strings[] property of the StringList.


HTH
DragonSlayer
0
Migrating Your Company's PCs

To keep pace with competitors, businesses must keep employees productive, and that means providing them with the latest technology. This document provides the tips and tricks you need to help you migrate an outdated PC fleet to new desktops, laptops, and tablets.

 
LVL 9

Expert Comment

by:ITugay
ID: 7018052
Hi charles_ebs,

be carefull with files bigger then 2GB.

-------
Igor.
0
 

Author Comment

by:charles_ebs
ID: 7024898
Found the bug.

The routine reads the file in 128-byte chunks.   The conditions under which the bug occured was if the 1st byte read into the buffer during the last read [that returned data] from the file happened to be a LF character.  If so, then it would keep returning phantom blank lines forever.

Here's the patched code section, with a comment marking the line that I added (look for the string "BUGFIX"):

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      If (Data.ReadCount > 0) then {BUGFIX C. Boling 5/21/01.  Added this conditional; otherwise, if 1st char in buffer happens to be #10 at eof, it would keep returning blank lines indefinitely.}
        CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;
 
0
 
LVL 6

Accepted Solution

by:
Mindphaser earned 0 total points
ID: 7025256
Charles

Thanks for the solution!

I will refund your points and move this to PAQ.

** Mindphaser - Community Support Moderator **
0

Featured Post

ScreenConnect 6.0 Free Trial

Explore all the enhancements in one game-changing release, ScreenConnect 6.0, based on partner feedback. New features include a redesigned UI, app configurations and chat acknowledgement to improve customer engagement!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

This article explains how to create forms/units independent of other forms/units object names in a delphi project. Have you ever created a form for user input in a Delphi project and then had the need to have that same form in a other Delphi proj…
In this tutorial I will show you how to use the Windows Speech API in Delphi. I will only cover basic functions such as text to speech and controlling the speed of the speech. SAPI Installation First you need to install the SAPI type library, th…
This Micro Tutorial will teach you how to censor certain areas of your screen. The example in this video will show a little boy's face being blurred. This will be demonstrated using Adobe Premiere Pro CS6.
Nobody understands Phishing better than an anti-spam company. That’s why we are providing Phishing Awareness Training to our customers. According to a report by Verizon, only 3% of targeted users report malicious emails to management. With compan…

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question