Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Reading UNIX format (LF delimited) text files - BUG

Posted on 2002-05-17
6
Medium Priority
?
195 Views
Last Modified: 2010-04-04
The driver for reading UNIX files in my previously asked question (qid=20295121) apparently has a little bug in it.  On certain files it will get to the end, then rather than properly detecting EOF, a ReadLn will forever keep returning blank lines.

I suspect it has to do with the last line coming out to the perfect length in the read buffer or something.  Below is the code: can anyone spot the problem?   I have two short sample files (1 good 1 bad) ready for testing.

=======================================================
unit StreamFile;
{
Use AssignStreamFile instead of AssignFile, and you can
read either CRLF *or* LF delimited files.


        Unix Stream File Interface
        Copyright 1996 by John Miano Software
        miano@worldnet.att.net

        5/1/2002 C. Boling  Status vars not used; removed references to prevent compiler warnings.
                            Also, remove any CR's on Read.
}
interface

Uses
  SysUtils ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;

implementation

Const
  BufferSize = 128 ;

Type
  TStreamBuffer = Array [1..High (Integer)] of Char ;
  TStreamBufferPointer = ^TStreamBuffer ;

  TStreamFileRecord = Record
    Case Integer Of
      1:
        (
          Filehandle : Integer ;
          Buffer : TStreamBufferPointer ;
          BufferOffset : Integer ;
          ReadCount : Integer ;
        ) ;
      2:
        (
          Dummy : Array [1 .. 32] Of Char
        )
    End ;


Function StreamFileOpen (var F : TTextRec) : Integer ;
  //Var
    //Status : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    GetMem (Buffer, BufferSize) ;
    Case F.Mode Of
      fmInput:
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone) ;
      fmOutput:
        FileHandle := FileCreate (StrPas (F.Name)) ;
      fmInOut:
        Begin
        FileHandle := FileOpen (StrPas (F.Name), fmShareDenyNone Or
fmOpenWrite or fmOpenRead) ;
        If FileHandle <> -1 Then
          //status := FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
          FileSeek (FileHandle, 0, 2) ; { Move to end of file. }
        F.Mode := fmOutput ;
        End ;
      End ;
    BufferOffset := 0 ;
    ReadCount := 0 ;
    F.BufEnd := 0 ;  { If this is not here it thinks we are at eof. }
    If FileHandle = -1 Then
      Result := -1
    Else
      Result := 0 ;
    End ;
  End ;

Function StreamFileInOut (var F : TTextRec) : Integer ;
  Procedure Read (var Data : TStreamFileRecord) ;
    Procedure CopyData ;
      Begin
      While (F.BufEnd < Sizeof (F.Buffer) - 2)
            And (Data.BufferOffset <= Data.ReadCount)
            And (Data.Buffer [Data.BufferOffset] <> #10) Do
        Begin
        {start 5-1-2002}
        //F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
        //Inc (Data.BufferOffset) ;
        //Inc (F.BufEnd) ;
        if Data.Buffer [Data.BufferOffset] <> #13 then
        begin
          F.Buffer [F.BufEnd] := Data.Buffer^ [Data.BufferOffset] ;
          Inc (F.BufEnd) ;
        end;
        Inc (Data.BufferOffset) ;
        {end 5-1-2002}
        End ;
      If Data.Buffer [Data.BufferOffset] = #10 Then
        Begin
        F.Buffer [F.BufEnd] := #13 ;
        Inc (F.BufEnd) ;
        F.Buffer [F.BufEnd] := #10 ;
        Inc (F.BufEnd) ;
        Inc (Data.BufferOffset) ;
        End ;
      End ;

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;

  Procedure Write (var Data : TStreamFileRecord) ;
    Var
      //Status : Integer ;
      Destination : Integer ;
      II : Integer ;
    Begin
    With TStreamFileRecord (F.UserData) Do
      Begin
      Destination := 0 ;
      For II := 0 To F.BufPos - 1 Do
        Begin
        If F.Buffer [II] <> #13 Then
          Begin
          Inc (Destination) ;
          Buffer^[Destination] := F.Buffer [II] ;
          End ;
        End ;
      //Status := FileWrite (FileHandle, Buffer^, Destination) ;
      FileWrite (FileHandle, Buffer^, Destination) ;
      F.BufPos := 0 ;
      Result := 0 ;
      End ;
    End ;
  Begin
  Case F.Mode Of
    fmInput:
      Read (TStreamFileRecord (F.UserData)) ;
    fmOutput:
      Write (TStreamFileRecord (F.UserData)) ;
    End ;
  End ;

Function StreamFileFlush (var F : TTextRec) : Integer ;
  Begin
  Result := 0 ;
  End ;

Function StreamFileClose (var F : TTextRec) : Integer ;
  Begin
  With TStreamFileRecord (F.UserData) Do
    Begin
    FreeMem (Buffer) ;
    FileClose (FileHandle) ;
    End ;
  Result := 0 ;
  End ;

Procedure AssignStreamFile (var F : Text ; Filename : String) ;
  Begin
  With TTextRec (F) Do
    Begin
    Mode := fmClosed ;
    BufPtr := @Buffer ;
    BufSize := Sizeof (Buffer) ;
    OpenFunc := @StreamFileOpen ;
    InOutFunc := @StreamFileInOut ;
    FlushFunc := @StreamFileFlush ;
    CloseFunc := @StreamFileClose ;
    StrPLCopy (Name, FileName, Sizeof(Name) - 1) ;
    End ;
  End ;
end.

0
Comment
Question by:charles_ebs
6 Comments
 
LVL 34

Expert Comment

by:Slick812
ID: 7017127
seems like an easier way would be to load a FileStream into a String variable and then use the AdjustLineBreaks( ) function on that string
0
 

Author Comment

by:charles_ebs
ID: 7017136
...unless you have a multi-gigabyte file and don't have that much RAM. :-)
0
 
LVL 14

Expert Comment

by:DragonSlayer
ID: 7017805
You should consider loading your text file into a TStringList, it will detect Unix/Windows linebreaks correctly (again, as what charles said, you shouldn't do that if you have a HUGE file).

You can then read it back line by line using the Strings[] property of the StringList.


HTH
DragonSlayer
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 9

Expert Comment

by:ITugay
ID: 7018052
Hi charles_ebs,

be carefull with files bigger then 2GB.

-------
Igor.
0
 

Author Comment

by:charles_ebs
ID: 7024898
Found the bug.

The routine reads the file in 128-byte chunks.   The conditions under which the bug occured was if the 1st byte read into the buffer during the last read [that returned data] from the file happened to be a LF character.  If so, then it would keep returning phantom blank lines forever.

Here's the patched code section, with a comment marking the line that I added (look for the string "BUGFIX"):

    Begin

    F.BufEnd := 0 ;
    F.BufPos := 0 ;
    F.Buffer := '' ;
    Repeat
      Begin
      If (Data.ReadCount = 0) Or (Data.BufferOffset > Data.ReadCount) Then
        Begin
        Data.BufferOffset := 1 ;
        Data.ReadCount := FileRead (Data.FileHandle, Data.Buffer^, BufferSize)
;
        End ;
      If (Data.ReadCount > 0) then {BUGFIX C. Boling 5/21/01.  Added this conditional; otherwise, if 1st char in buffer happens to be #10 at eof, it would keep returning blank lines indefinitely.}
        CopyData ;
      End Until (Data.ReadCount = 0)
                Or (F.BufEnd >= Sizeof (F.Buffer) - 2) ;
    Result := 0 ;
    End ;
 
0
 
LVL 6

Accepted Solution

by:
Mindphaser earned 0 total points
ID: 7025256
Charles

Thanks for the solution!

I will refund your points and move this to PAQ.

** Mindphaser - Community Support Moderator **
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
Hello everybody This Article will show you how to validate number with TEdit control, What's the TEdit control? TEdit is a standard Windows edit control on a form, it allows to user to write, read and copy/paste single line of text. Usua…
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…
Suggested Courses

824 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question