how to read 10 line from the last logfile in text file formatted, asap

hi expert.
i wanna ask some question about text files.

i got a logfile with 2 mb size and it can be increase in anytime.
i want to show all the logfile in text file formatted but it takes time for showing all that logfile. so i've decided to take at least 10 or 15 lines from the last of the logfile. and if the logfile was changing that changing should be included in a logfile too.
clue:
-open logfile in text file formatted
-show all the logfile in text file formatted
-show at least 10 or 15 from the last logfile
- if there's a changing between showing the text file, that changing should be include in the logfile
-and must be using a low resources and speed is critical  
with this i had include the logfile sample.
can u help me with this? thanks expert for helping me. i need this source asap.

sample logfile
11072002 121100  Debug level       = 3
11072002 121100  Debug Filename    = c:\PROGRA~1\MAX\log\r2_in.log
11072002 121100  Voice Resource    = 30
11072002 121100  Inter Digit Delay = 100
11072002 121100  DTI Cfg Filename  = c:\PROGRA~1\MAX\etc\r2in.cfg
11072002 121100  DTI Map Filename  = c:\PROGRA~1\MAX\etc\linemap.cfg
11072002 121100  TCP HostName      = localhost
11072002 121100  TCP Port          = 5000
11072002 121100  ISDN Cfg Filename =
11072002 121100  inifile=C:\PROGRAM FILES\MAX\etc\DTGC_IN.ini
11072002 121100  label = R2_IN
11072002 121100  [SYS] Init Tcp socket
11072002 121100  [SYS] Init VoiceBoard
11072002 121100  VC-Chn1 :InitVoice variable
11072002 121100  Open Voice resource dxxxB1C1, ts =0
11072002 121100  VC-Chn2 :InitVoice variable
11072002 121100  Open Voice resource dxxxB1C2, ts =1
11072002 121100  VC-Chn3 :InitVoice variable
11072002 121100  Open Voice resource dxxxB1C3, ts =2
11072002 121100  VC-Chn4 :InitVoice variable
11072002 121100  Open Voice resource dxxxB1C4, ts =3
11072002 121100  VC-Chn5 :InitVoice variable
11072002 121100  Open Voice resource dxxxB2C1, ts =4
denoxAsked:
Who is Participating?
 
SpideyModCommented:
All,
I am unlocking this question in preparation for cleanup.  I will return in 7 days to finalize this question.  Please leave any recommendations for the final state of this question, I will take all recommendations into consideration.  Failing any feedback, I may decide in 7 days to delete or PAQ this question with no refund.  Thanks.

SpideyMod
Community Support Moderator @Experts Exchange
0
 
thirdCommented:
try this,


unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, ComCtrls;

type
  TForm1 = class(TForm)
    RichEdit1: TRichEdit;
    OpenDialog1: TOpenDialog;
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.Button1Click(Sender: TObject);
const
  maxline = 10;
var
  F: TextFile;
  S, tempstr: string;
  i : integer;
begin
  tempstr := '';
  if OpenDialog1.Execute then            { Display Open dialog box }
  begin
    AssignFile(F, OpenDialog1.FileName); { File selected in dialog }
    Reset(F);
    for i := 1 to maxline do
    begin
      Readln(F, S);
      tempstr :=  tempstr + S + #13#10;
    end;
    RichEdit1.Clear;
    RichEdit1.Text := tempstr;
    CloseFile(F);
  end;
end;

end.
0
 
CrazyOneCommented:
third
If I am not mistaken denox wants to read the last 10 or 15 lines of the file. If I interpret your code correctly it is only reading the first few lines. :>)
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
thirdCommented:
oppps.... ;-)
0
 
Hamlet081299Commented:
Try this...

procedure TForm1.Button1Click(Sender: TObject);
const
  LINES = 15;
var
  i: integer;
  ch: char;
  s: String;
begin

  with TFileStream.Create('Unit1.pas', fmOpenRead) do try
    Seek(-1, soFromEnd);
    for i := 1 to LINES + 1 do begin
      while (Read(ch, 1) = 1) and (Seek(-2, soFromCurrent) > 0) and (ch <> #10) do;
    end;
    Seek(2, soFromCurrent);

    s := '';
    while (Read(ch, 1) = 1) do
      s := s + ch;

    Memo1.Text := s;
  finally
    Free;
  end;
end;
0
 
MadshiCommented:
My suggestion is this:

(1) Open the log file as a memory mapped file like this (the code is written from my head and NOT tested):

var file_, map   : dword;
    view         : pchar;
    sizeL, sizeH : dword;
    size64       : int64;
begin
  file_ := CreateFile('c:\your.log', GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, 0, 0);
  if file_ <> INVALID_HANDLE_VALUE then begin
    sizeL := GetFileSize(file_, @sizeH);
    size64 := int64(sizeH) shl 32 + sizeL;
    map := CreateFileMapping(file_, nil, PAGE_READWRITE, 0, 0, nil);
    if map <> 0 then begin
      view := MapViewOfFile(map, FILE_MAP_READ, 0, 0, 0);
      if view <> nil then begin
        [...]  // here you have the file in memory at pointer "view" with the length "size64"
        UnmapViewOfFile(view);
      end;
      CloseHandle(map);
    end;
    CloseHandle(file_);
  end;
end;

Please note that the file is not yet loaded into memory! So the opening as above doesn't cost any noticable time!

(2) Now you can browse through the memory. In the moment when you access the file memory opened above, Windows automatically loads the requested bytes from the file. This is very fast and optimized by Windows. The find the last 10 lines, search backwards for #13#10. To search backwards you can e.g. use my function http://help.madshi.net/Data/StringSearch.htm#PosPChar for this purpose. It's contained in my free package "madBasic", which you can download from my homepage for free including sources.

Sorry, have no time to write more sources or to test them...

Regards, Madshi.
0
 
mpootsCommented:
Listening
0
 
Slick812Commented:
hello denox, since you don't really seem to need a definite number of lines so I did not do a line count I just used the average line length time 15


procedure TForm1.buttton_ResdTextClick(Sender: TObject);
var
FileName: String;
Sfile1, BytesRead: Cardinal;
AryChar: Array[0..678] of Char;
{set the Array length to the average line length times 15}
ScanPc: PChar;
begin
Filename := 'C:\Stuff\Scanner Install log.txt';
Sfile1 := CreateFile(PChar(FileName),GENERIC_READ,FILE_SHARE_READ,
          nil,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0);
try
if Sfile1 > 0 then
  begin
  SetFilePointer(Sfile1, -SizeOf(AryChar), nil, FILE_END);
  if not ReadFile(Sfile1,AryChar,SizeOf(AryChar),BytesRead,nil) then
    begin
    ShowMessage('There was a read error');
    Exit;
    end;
  ScanPc := StrScan(AryChar, #10)+1;
  Memo1.Text := ScanPc;
  end;
finally
CloseHandle(SFile1);
end;
end;

- - - - - - - - -  - - - -  - -
I guess this might be a touch faster ?
0
 
DumaniCommented:
I would use TStringList...

procedure ShowLines(LineCount: Integer);
var
  ml: TStringList;
  i: integer;
begin
  ml := TStringList.Create;
  try
    ml.LoadFromFile('C:\mylog.log');
    for i := (ml.Count - 1) - LineCount to ml.Count - 1 do
    begin
      Memo1.Lines.Add(ml[i]);
    end;
  finally
    ml.Free;
  end;
end;

this procedure will show you last linecount lines.

Hope that works...
0
 
Lee_NoverCommented:
madshi nice method with mmf
hope the kids value the speed over complexity of the code :)
0
 
MadshiCommented:
:-)  Thank you...
0
 
smurffCommented:
listening
0
 
Hamlet081299Commented:
I believe my example was the first to meet the criteria, so to "encourage" the querent to make a decision I will propose it as an answer (after a little bit of tidying...

function LoadLogEnd(const LogName: String; Lines: integer): String;
var
  i: integer;
  ch: char;
begin
  with TFileStream.Create(LogName, fmOpenRead) do try
    Seek(-1, soFromEnd); // go to end
    for i := 1 to Lines + 1 do
      while (Read(ch, 1) = 1) and (Seek(-2, soFromCurrent) > 0)
      and (ch <> #10) do; // read backwards looking for LF's
   Seek(2, soFromCurrent); // Reposition after LF

   Result := '';
   while (Read(ch, 1) = 1) do Result := Result + ch;
 finally
   Free;
 end;
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  Memo1.Text := LoadLogEnd('Unit1.pas', 15);
end;

0
 
Lee_NoverCommented:
instead of: while (Read(ch, 1) = 1) do Result := Result + ch;

you should do :

SetLength(Result, Size - Position);
SetLength(Result, Read(PChar(Result)^, Size - Position));

(I'd also create tmp local vars for the string length .. check what it does when you call Size and Position and you'll see why ... I just have to optimize it all :)
0
 
pedeCommented:
I made something like this a while ago. This was made with speed in mind, and I believe it is very fast (I dont know how it will compare to Madshi's memory mapped files, but that could be tested of course).

To get the speed I wanted I assumed that the average line was not over 100 characters. I know my data so this assumption is safe. The function will return more lines than the requested count, but its just a matter of only using the number you want. I believe its much faster to just get a larger number of lines, than to scan backwards to get the exact number.

(sorry for the danish comments in the code :o)

function GetLoglines(Fil: string; Cnt: integer; List: TStringList): integer;
const
  MINBUFLEN = 32768;
  ESTIMATEMAXLINELEN = 100;
var
  F             : file;
  Pos           : integer;
  P             : integer;
  Buf           : array of byte;
  BufLen        : integer;
  LinesFound    : integer;
  BytesRead     : integer;
  TmpStr        : string;
  LineEnd       : integer;
  LineStart     : integer;
  OldFileMode   : integer;
begin
  try
    BufLen := Cnt * ESTIMATEMAXLINELEN;
    { Vi satser på at alle liner kan være i bufferen, ellers er worst case at der misses et par stykker (UNLIKELY) }
    { Minimum buffer er MINBUFLEN, der kan kun misses linier hvis man vælger rigtig mange og de er rigtig lange (> 100 i snit) }
    if (BufLen < MINBUFLEN) then BufLen := MINBUFLEN;
    SetLength(Buf, BufLen);

    OldFileMode := FileMode;
    FileMode := 0;
    AssignFile(F, Fil);
    Reset(F, 1);
    FileMode := OldFileMode;

    Pos := FileSize(F)-Length(Buf);
    if (Pos < 0) then Pos := 0;
    LinesFound := 0;
    Seek(F, Pos);
    BlockRead(F, Buf[0], Length(Buf), BytesRead);
    P := BytesRead-1;
    { står på #10 ved start, med mindre filen er tom eller forkert format, hop hen til #13 }
    Dec(P);
    while (P > 0) and (LinesFound < Cnt) do begin
      LineEnd := P;
      while (P > 0) and (Buf[P] <> 10) do Dec(P);
      { only add one if condition [Buf[P] <> 10] caused loop to stop (then we are on a #13), not if P = 0 }
      if (P = 0) then
        LineStart := P
      else
        LineStart := P+1;
      Dec(P);
      SetString(TmpStr, pchar(@Buf[LineStart]), LineEnd - LineStart);
      List.Add(TmpStr);
      Inc(LinesFound);
    end;
    CloseFile(F);
    Result := List.Count;
  except
    on e: exception do begin
      Result := -1;
      try
        List.Text := E.Message + ' [ '+Fil+' ] ';
      except
      end;
    end;
  end;
end;
0
 
Slick812Commented:
pede, just a thought, , , ,
you may want to look at a couple of utility functions in the Classes unit that can be helpful, consider the

LineStart(Buffer, BufPos: PChar): PChar;

and the

ExtractStrings(Separators, WhiteSpace: TSysCharSet; Content: PChar; Strings: TStrings): Integer;
0
 
Hamlet081299Commented:
Denox - You have not even commented since your original question.  If you need clarification on the comments given or feel that they do not meet your requirements please let us know!!!

This is a relatively simple problem.  The initial problem resulted from reading a LARGE text file into a memo.

This is going to be slow for two reasons...
1. The size of the file
2. The fact that TMemo's are abysmally slow when you load a lot of data into them.

In fact I suspect that #2 was probably the worst of it.

The solution requires a smaller number of lines to be loaded.  This can be easily achieved by going to the end of the file and looking back until the nth line end is found.

And pardon me for being obvious and simple, but that should prove fast enough.

Some developers try far too hard to be clever.  The result being overly complex code that noone else will ever understand, but that for some reason it panders to their own ego.

I've been plenty guilty of it in the past myself.

No disrespect meant to other peoples suggestions, but can we please just move on?
0
 
denoxAuthor Commented:
to all the expert

thank you for helping me with the question
but the expert in here only help me to get the 10 or 15 lines from the last file.
but how about if there's a changing in a logfile when i read the file, how to include the changing and add into the logfile that i was reading it on ?
thank you for helping me

0
 
Hamlet081299Commented:
There's a number of options, depending on how "realtime" you want it to be....

You could put in a file hook to monitor the status of the file.

... or ...

You could check the file every 30 seconds (or whatever) and if the timestamp has changed since the last refresh then call the load procedure again.
0
 
Lee_NoverCommented:
as Hamlet suggested a change notification would help you monitor that
check out the FindFirstChangeNotification and FindNextChangeNotification functions
0
 
Lee_NoverCommented:
> but how about if there's a changing in a logfile when i read the file <

about that .. there's really small chance of that happening
you could also lock the file for writing while you're reading it
but reading the last 20 lines (about 400 bytes) would take less than 20 ms to read
0
 
Slick812Commented:
I guess I understand what you want to do now, Here some code to reload any changes to a Text (Log) file and show the last 12 lines of code in a memo. This uses another Thread to monitor the changes in the Folder where the Log file is located.



private
    { Private declarations }
    function SizeOFile(FileName:String):Int64;
    function GetLastLines(Lines: Word): PChar;

var
  Form1: TForm1;
  ChangeH, hThread: THandle;
  LogStr1: String;
{LogStr1 will have the entire Log file in it}
  LogSize1: Integer;

implementation



function TForm1.SizeOFile(FileName:String):Int64;
var
FindHandle:HWND;
FindData: TWin32FindData;
ErrorMode: Word;
begin
ErrorMode := SetErrorMode(SEM_FailCriticalErrors);
FindHandle := FindFirstFile(PChar(FileName), FindData);
SetErrorMode(ErrorMode);
if FindHandle <> INVALID_HANDLE_VALUE then
  begin
  Result := (FindData.nFileSizeHigh * MAXDWORD) + FindData.nFileSizeLow;
  Windows.FindClose(findHandle);
  end else Result := -1;
end;

function TForm1.GetLastLines(Lines: Word): PChar;
var
CharPos: PChar;
NumLines: Word;
begin
Result := #0;
CharPos := @LogStr1[Length(LogStr1)];
  NumLines := 0;
  while (CharPos <> @LogStr1[1]) and (NumLines < Lines) do
    begin
    CharPos := LineStart(@LogStr1[1], CharPos-2);
    Inc(NumLines);
    end;
Result := CharPos;
end;

function ThreadFunc(Parameter: Pointer): Integer; stdcall;
var
FileName, AddText: String;
SFile1: THandle;
NewSize: Int64;
BytesRead: Cardinal;
begin
Result := 0;
ChangeH := FindFirstChangeNotification('C:\Stuff', False, FILE_NOTIFY_CHANGE_SIZE);
sleep(40);
while true do
  begin
  if WaitForSingleObject(ChangeH, INFINITE) = WAIT_OBJECT_0 then
    begin
    Filename := 'C:\Stuff\Scanner Install log.txt';
    NewSize := Form1.SizeOFile(FileName);
    if NewSize > LogSize1 then
      begin
      NewSize := NewSize-LogSize1;
      Sfile1 := CreateFile(PChar(FileName),GENERIC_READ,FILE_SHARE_READ,
          nil,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0);
      try
        if Sfile1 > 0 then
          begin
          SetLength(AddText, NewSize);
          SetFilePointer(Sfile1, -NewSize, nil, FILE_END);
          if ReadFile(Sfile1,Pointer(AddText)^,NewSize,BytesRead,nil) and
             (BytesRead > 1) then
            begin
            LogStr1 := LogStr1 + AddText;
            LogSize1 := LogSize1+ NewSize;
            Form1.Memo1.Text := Form1.GetLastLines(12);
            end;
          end;
      finally
      CloseHandle(SFile1);
      end;
    end;

  end;
FindNextChangeNotification(ChangeH);
end;
EndThread(Result);
end;


procedure TForm1.FormCreate(Sender: TObject);
begin
hThread := 0;
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
if hThread > 0 then
  begin
  FindCloseChangeNotification(ChangeH);
  CloseHandle(hThread);
  end;
end;


procedure TForm1.button_StartTextClick(Sender: TObject);
var
FileName: String;
Sfile1, BytesRead, ChangeH, ThreadId: Cardinal;
ScanPc: PChar;

begin
{this button click Starts the File read and starts the Thread}
Filename := 'C:\Stuff\Scanner Install log.txt';
Sfile1 := CreateFile(PChar(FileName),GENERIC_READ,FILE_SHARE_READ,
          nil,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL or FILE_FLAG_SEQUENTIAL_SCAN,0);
try
if Sfile1 > 0 then
  begin
  LogSize1 := SetFilePointer(Sfile1, 0, nil, FILE_END);
  SetLength(LogStr1, LogSize1);
  SetFilePointer(Sfile1, 0, nil, FILE_BEGIN);
  if not ReadFile(Sfile1, Pointer(LogStr1)^, LogSize1,BytesRead,nil) or
     (BytesRead < 3) then
    begin
    ShowMessage('There was a read error');
    Exit;
    end;
  Memo1.Text := GetLastLines(12);
  end;
finally
CloseHandle(SFile1);
end;
if Sfile1 > 0 then
  begin
  hThread := BeginThread(nil, 0, @ThreadFunc, nil, 0, ThreadId);
  if hThread > 0 then
  button_EndThread.Enabled := True;
  end;
end;

procedure TForm1.sbut_EndThreadClick(Sender: TObject);
begin
FindCloseChangeNotification(ChangeH);
CloseHandle(hThread);
end;

- - - - - - - - - - - - - - - - - - - - - - - -

didn't have time to comment very much, ask questions if it's unclear
0
 
denoxAuthor Commented:
to all expert especially slick812

hi expert i've been reading to all your answer about my question.
and i find slick812 answers its the closest answer that i needed for my problems
but i still got some problems here with your answer and what i want for my problem is just like this:

this is the scenes :
1.browse the log file in any folder and any directory
2.read all log file before changing  (displayed all log file)
while not eof (log file) do
begin
   3.get 15 line from the last line of the log file (only show the 15 line from the previous log file )
   4.if there's a changing in the log file, the changing will be adding in line 16 untill all the log file
      add in.
end;
5. should be able to open or browse other log file  

all the program should be flexibel and not static also time and speed is critical
thank you for all the expert helps in my problems.
sincerely,denox
0
 
denoxAuthor Commented:
to all expert especially slick812

hi expert i've been reading to all your answer about my question.
and i find slick812 answers its the closest answer that i needed for my problems
but i still got some problems here with your answer and what i want for my problem is just like this:

this is the scenes :
1.browse the log file in any folder and any directory
2.read all log file before changing  (displayed all log file)
while not eof (log file) do
begin
   3.get 15 line from the last line of the log file (only show the 15 line from the previous log file )
   4.if there's a changing in the log file, the changing will be adding in line 16 untill all the log file
      add in.
end;
5. should be able to open or browse other log file  

all the program should be flexibel and not static also time and speed is critical
thank you for all the expert helps in my problems.
sincerely,denox
0
 
Slick812Commented:
it seems that you could use the code I have already given you to do what you want, for instance , about the 1 ".browse the log file in any folder and any directory". . .  . .  in the button_StartTextClick procedure just use Filename := ToAnyFileYouWant

about the 2 "2.read all log file before changing  (displayed all log file", , , , , , the   LogStr1 string variable has the ENTIRE log file in it, ALL the time, so you can Display it (or part of it) ant where you want (a TRichEdit).

about the 3 "get 15 line from the last line of the log file (only show the 15 line from the previous log file " is already there,


about the 4 "if there's a changing in the log file, the changing will be adding in line 16 untill all the log file
     add in", , , ,  , the GetLastLines(Lines: Word): PChar function allows you to get any number of lines you need

what do you need help with ?

 - - - - - - - - - - - - - - - - - - - - - -  -

and you might use this code instead for the ThreadFunc


function ThreadFunc(Parameter: Pointer): Integer; stdcall;
var
FileName, AddText: String;
SFile1: THandle;
NewSize: Int64;
BytesRead: Cardinal;
begin
Result := 0;
ChangeH := FindFirstChangeNotification('C:\Stuff', False, FILE_NOTIFY_CHANGE_SIZE);
if ChangeH < 2 then
  begin
  Form1.Memo1.Text := 'Change Notification Failed, could not monitor Folder';
  EndThread(Result);
  end;
sleep(40);
while true do
begin
if WaitForSingleObject(ChangeH, INFINITE) = WAIT_OBJECT_0 then
  begin
  Filename := 'C:\Stuff\Scanner Install log.txt';
  NewSize := Form1.SizeOFile(FileName);
  if NewSize > LogSize1 then
    begin
    NewSize := NewSize-LogSize1;
    Sfile1 := CreateFile(PChar(FileName),GENERIC_READ,FILE_SHARE_READ or FILE_SHARE_WRITE,
          nil,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0);
    try
      if Sfile1 > 0 then
        begin
        SetLength(AddText, NewSize);
        SetFilePointer(Sfile1, -NewSize, nil, FILE_END);
        if ReadFile(Sfile1,Pointer(AddText)^,NewSize,BytesRead,nil) and
           (BytesRead > 1) then
           begin
           LogStr1 := LogStr1 + AddText;
           LogSize1 := LogSize1+ NewSize;
           Form1.Memo1.Text := Form1.GetLastLines(12);
           end;
        end;
      finally
      CloseHandle(SFile1);
      end;
    end;

  end;
if not FindNextChangeNotification(ChangeH) then
    begin
    Form1.Memo1.Text := 'Change Notification Failed, could not monitor Folder';
    FindCloseChangeNotification(ChangeH);
    Break;
    end;
end;

EndThread(Result);
end;
0
 
Slick812Commented:
denox, is what you need a way to monitor the changes to more than one log file? ? If you set 3 or 4 log files then whenever any of them change you will see the change? You say something about "Display" and "Change" the logfile, if the log files are 2 megs, it seems like more text than a person could deal with. And I don't understand your "only show the 15 line from the previous log file", Are you talking about many TRichEdits for display or just one?
0
 
Slick812Commented:
denox, do you need something else?, would you like to see the code for monitoring more than one log file with a waitforMutipleObjects ?
0
 
BudaCommented:
I would use stringlist to load the data from file as proposed by Dumani. For adding data I would use following:

procedure Log(data:string);
const
 LINES_COUNT = 15;
var
 F: TextFile;
begin
 //save the data into log file
 AssignFile(F,'log.txt');
 Append(F);
 WriteLn(F,data);
 Flush(F);
 CloseFile(F);
 
  //add to the end of memo
  Memo1.Lines.Add(Data);

  //keep count of lines constant
  while Memo1.Lines.Count > LINES_COUNT do
    Memo1.Lines.Delete(0);
end;
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.