Solved

[123.234.435.234] err <-- Extract Numbers from between brackets

Posted on 2006-06-23
9
291 Views
Last Modified: 2012-05-05
Hello All;

  I need to go through A LOT of Mail Server Log Files.

This is what the string will look like:

20060623 055059 127.0.0.1       SMTPD (002700E8) [210.86.39.32] ERR my-domain.com invalid user
(Of course the my-domain.com, is not Our Domain, just changed it for here.

And I need a way to extract out the BAD IP Address.
Which as you can see is located in the 5th Column   [210.86.39.32]
I need to then put the Bad IP Address's into another TMemo.

Our log files are FULL of Bad IP Address, The code must be very robust.

Thanks all;
Carrzkiss
0
Comment
Question by:Wayne Barron
  • 5
  • 4
9 Comments
 
LVL 26

Expert Comment

by:Russell Libby
ID: 16973171
Wayne,

You could use the following to extract the ip addr from each line. It does simple verification by checking the string for the following:

- starts with [
- contains numbers
- ends with ]
- has 3 "." chars embedded in the numeric string

Not that it can't be fooled, eg [3..2.22], but it should be fine for your log parsing. If you load your log file into a TStrings or TStringList, then you can just iterate each line and pass it to the function. If no IP addresses are matched in the (line) string, then a blank string will be returned, otherwise the ip address is returned. if you have questions, let me know.

Regards,
Russell

function ExtractIPAddr(Value: String): String;
var  lpszStart:     PChar;
     lpszEnd:       PChar;
     lpszCheck:     PChar;
     dwCount:       Integer;
begin

  // Set default result
  SetLength(result, 0);

  // Check length
  if (Length(Value) > 0) then
  begin
     // Scan for [
     lpszStart:=StrScan(Pointer(Value), '[');
     // Check result
     while Assigned(lpszStart) do
     begin
        // Push past the [
        Inc(lpszStart);
        // Scan for ]
        lpszEnd:=StrScan(lpszStart, ']');
        // Check result
        if Assigned(lpszEnd) then
        begin
           // Set check string
           lpszCheck:=lpszStart;
           // Set counter for .
           dwCount:=0;
           // Check embedded string
           while (lpszCheck < lpszEnd) do
           begin
              // Check for .
              if (lpszCheck^ = '.') then
                 // Increment the counter
                 Inc(dwCount)
              // Check for numerics
              else if not(lpszCheck^ in ['0'..'9']) then
                 // Break, not a numeric
                 break;
              Inc(lpszCheck);
           end;
           // Check last char for numeric
           if (dwCount = 3) and (lpszCheck^ = ']') then
           begin
              // Found result
              SetString(result, lpszStart, lpszEnd-lpszStart);
              // Done processing
              break;
           end;
           // Scan again
           lpszStart:=StrScan(lpszEnd, '[');
        end
        else
           // Can't match
           lpszStart:=nil;
     end;
  end;

end;

  // Example
  ShowMessage(ExtractIPAddr('20060623 055059 127.0.0.1     SMTPD (002700E8) [210.86.39.32] ERR my-domain.com invalid user'));



0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 16973223
Hello Russell;

How would I implement grabbing the IP Address's from between the [] ERR
And listing them in the 2ndMemo.
And not adding in ""Duplicate"" entries?
As the files will have sections where the Entries are going to multiple
invalid users.
(Which is a complete pain, gives us names that we do not want to use :)  )

Thanks
Wayne
0
 
LVL 26

Accepted Solution

by:
Russell Libby earned 300 total points
ID: 16973272
Example provided, source first, dfm follows:

Russell

---

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs,
  StdCtrls;

type
  TForm1 = class(TForm)
    Button1: TButton;
    Memo1: TMemo;
    Memo2: TMemo;
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation
{$R *.DFM}

function ExtractIPAddr(Value: String): String;
var  lpszStart:     PChar;
     lpszEnd:       PChar;
     lpszCheck:     PChar;
     dwCount:       Integer;
begin

  // Set default result
  SetLength(result, 0);

  // Check length
  if (Length(Value) > 0) then
  begin
     // Scan for [
     lpszStart:=StrScan(Pointer(Value), '[');
     // Check result
     while Assigned(lpszStart) do
     begin
        // Push past the [
        Inc(lpszStart);
        // Scan for ]
        lpszEnd:=StrScan(lpszStart, ']');
        // Check result
        if Assigned(lpszEnd) then
        begin
           // Set check string
           lpszCheck:=lpszStart;
           // Set counter for .
           dwCount:=0;
           // Check embedded string
           while (lpszCheck < lpszEnd) do
           begin
              // Check for .
              if (lpszCheck^ = '.') then
                 // Increment the counter
                 Inc(dwCount)
              // Check for numerics
              else if not(lpszCheck^ in ['0'..'9']) then
                 // Break, not a numeric
                 break;
              Inc(lpszCheck);
           end;
           // Check last char for numeric
           if (dwCount = 3) and (lpszCheck^ = ']') then
           begin
              // Found result
              SetString(result, lpszStart, lpszEnd-lpszStart);
              // Done processing
              break;
           end;
           // Scan again
           lpszStart:=StrScan(lpszEnd, '[');
        end
        else
           // Can't match
           lpszStart:=nil;
     end;
  end;

end;

procedure TForm1.Button1Click(Sender: TObject);
var  listAddrs:     TStringList;
     szIp:          String;
     dwIndex:       Integer;
begin

  Memo1.Lines.BeginUpdate;
  try
     listAddrs:=TStringList.Create;
     try
        listAddrs.Sorted:=True;
        listAddrs.Duplicates:=dupIgnore;
        for dwIndex:=0 to Pred(Memo1.Lines.Count) do
        begin
           szIp:=ExtractIPAddr(Memo1.Lines[dwIndex]);
           if (Length(szIp) > 0) then listAddrs.Add(szIp);
        end;
        Memo2.Lines.Assign(listAddrs);
     finally
        listAddrs.Free;
     end;
  finally
     Memo1.Lines.EndUpdate;
  end;

end;

end.

---

object Form1: TForm1
  Left = 231
  Top = 127
  Width = 724
  Height = 366
  Caption = 'Form1'
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'MS Sans Serif'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 96
  TextHeight = 13
  object Button1: TButton
    Left = 8
    Top = 8
    Width = 75
    Height = 25
    Caption = 'Process'
    TabOrder = 0
    OnClick = Button1Click
  end
  object Memo1: TMemo
    Left = 8
    Top = 40
    Width = 465
    Height = 285
    Lines.Strings = (
      'Hello All;'
      ''
      'I need to go through A LOT of Mail Server Log Files.'
      ''
      'This is what the string will look like:'
      ''
     
        '20060623 055059 127.0.0.1 SMTPD (002700E8) [210.86.39.32] ERR my' +
        '-domain.com invalid '
      'user '
     
        '(Of course the my-domain.com, is not Our Domain, just changed it' +
        ' for here.'
      ''
      'And I need a way to extract out the BAD IP Address.'
      'Which as you can see is located in the 5th Column [210.86.39.32]'
      'I need to then put the Bad IP Address'#39's into another TMemo.'
      ''
     
        'Our log files are FULL of Bad IP Address, The code must be very ' +
        'robust.'
      ''
      'Thanks all;'
      'Carrzkiss'
      'Hello All;'
      ''
      'I need to go through A LOT of Mail Server Log Files.'
      ''
      'This is what the string will look like:'
      ''
     
        '20060623 055059 127.0.0.1 SMTPD (002700E8) [210.86.39.32] ERR my' +
        '-domain.com invalid '
      'user '
     
        '(Of course the my-domain.com, is not Our Domain, just changed it' +
        ' for here.'
      ''
      'And I need a way to extract out the BAD IP Address.'
      'Which as you can see is located in the 5th Column [210.86.39.32]'
      'I need to then put the Bad IP Address'#39's into another TMemo.'
      ''
     
        'Our log files are FULL of Bad IP Address, The code must be very ' +
        'robust.'
      ''
      'Thanks all;'
      'Carrzkiss'
      'Hello All;'
      ''
      'I need to go through A LOT of Mail Server Log Files.'
      ''
      'This is what the string will look like:'
      ''
     
        '20060623 055059 127.0.0.1 SMTPD (002700E8) [210.86.39.32] ERR my' +
        '-domain.com invalid '
      'user '
     
        '(Of course the my-domain.com, is not Our Domain, just changed it' +
        ' for here.'
      ''
      'And I need a way to extract out the BAD IP Address.'
      'Which as you can see is located in the 5th Column [210.86.39.32]'
      'I need to then put the Bad IP Address'#39's into another TMemo.'
      ''
     
        'Our log files are FULL of Bad IP Address, The code must be very ' +
        'robust.'
      ''
      'Thanks all;'
      'Carrzkiss'
      'Hello All;'
      ''
      'I need to go through A LOT of Mail Server Log Files.'
      ''
      'This is what the string will look like:'
      ''
     
        '20060623 055059 127.0.0.1 SMTPD (002700E8) [210.86.39.32] ERR my' +
        '-domain.com invalid '
      'user '
     
        '(Of course the my-domain.com, is not Our Domain, just changed it' +
        ' for here.'
      ''
      'And I need a way to extract out the BAD IP Address.'
      'Which as you can see is located in the 5th Column [210.86.34.32]'
      'I need to then put the Bad IP Address'#39's into another TMemo.'
      ''
     
        'Our log files are FULL of Bad IP Address, The code must be very ' +
        'robust.'
      ''
      'Thanks all;'
      'Carrzkiss')
    ScrollBars = ssVertical
    TabOrder = 1
  end
  object Memo2: TMemo
    Left = 480
    Top = 40
    Width = 225
    Height = 285
    ScrollBars = ssVertical
    TabOrder = 2
  end
end
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 16973335
That is sweet.
Works like a charm, and list the IP Address in Numerical Order. Which is real cool.

Thank you once again. The code is easy to understand as well.
-------------
After see the way this code works, and how it arranges the IP Address in Numerical Order.
Got me to thinking about a site that I am doing for "Banned IP Address's"
The list is not "Numerically Ordered"

Do you think that you could do something like this?

Here is a segment of the HTML Code.

<tr><th rowspan=1>12.109.221.148  &nbsp;</td><th rowspan=1>&nbsp;255.255.255.255 </td><th rowspan=1><a href="http://www.dnsstuff.com/tools/whois.ch?ip=12.109.221.148"target="_blank">AT&T WorldNet Services</a></td><td><a href="http://att.com/worldnet/" target="_blank">AT&T</a></td></tr>

(Yes, you got it, we get in a LOT of spam)

Basically, you would need to look at the beginning of the line here:   <tr><th rowspan=1>12.109.221.148
And then loop through the entire list and arrange all the Numbers, "Without" corrupting the HTML Line String(s).
As that is what scares me about it, is "If" the code is not done correctly, then it could off-set the HTML Code
And cause it to split into other lines.

The Code above, is 1-Single Line
There is no double lines in the entire site, this is to help keep everything clean and neat for us to work in.

If this is something that you can do, let me know, and I will be more then happy to open another question
And point you to it.

Going to up the points here as well from 125 - 300
Thank you once again Russell.

Wayne
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 26

Expert Comment

by:Russell Libby
ID: 16973627

Probably best to switch to a regular expression parser at this point; and one that I am familiar with (having written it) that includes source can be download from:

http://users.adelphia.net/~rllibby/downloads/regexprex.zip
 
Then the following routine can be applied, just select the desired pattern to use:

uses
  ..., RegExprEx;

// Patterns for both the original and new question
const
  // Pattern for original question
  OldPattern = '\[(:0\d+\.\d+\.\d+.\d+)\] ERR';
  // Pattern for new question
  NewPattern = '<tr><th rowspan=1>(:0\d+\.\d+\.\d+\.\d+)!(</tr>)*</tr>';

var  listAddrs:     TStringList;
begin
 
  Memo1.Lines.BeginUpdate;
  try
     listAddrs:=TStringList.Create;
     try
        listAddrs.Sorted:=True;
        listAddrs.Duplicates:=dupIgnore;
        with TRegExpr.CreatePattern(NewPattern) do
        begin
           try
              Source:=PChar(Memo1.Lines.Text);
              if MatchFirst then
              begin
                 repeat
                    listAddrs.Add(BackReference[0]);
                 until not(MatchNext);
              end;
           finally
              Free;
           end;
        end;
        Memo2.Lines.Assign(listAddrs);
     finally
        listAddrs.Free;
     end;
  finally
     Memo1.Lines.EndUpdate;
  end;
 
end;
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 16973692
Hello Russell;

This is not quite what I had in mind.
I need to sort through the HTML Code, and Re-arrange (Organize) The IP Address's in Order
From Least to Greatest.
Example

234.54.56.129
125.34.57.123
241.45.23.12
24.345.54.92
221.23.14.15

In the above lines, they are not Ordered. I Would like (But is not a real big deal) to have them ordered
So that the ISP's will all be together, and the IP Address's will be much better to look through, as they are in Order.
So it would look something like this:

24.345.54.92
125.34.57.123
221.23.14.15
234.54.56.129
241.45.23.12
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 16973696
Sorry, It will still have the HTML Code wrapping it.
The HTML Code has to stay in place, and the [Lines] moved around.
Like a Virtual Washing Machine.
That moves the lines around until they are all Ordered in a Numbered Sequense.

Thank You
Wayne
0
 
LVL 26

Expert Comment

by:Russell Libby
ID: 16973702
So you are talking about modifying the html text so that the <tr> ... </tr> elements are ordered by the IP address specified within them?

Russell
0
 
LVL 30

Author Comment

by:Wayne Barron
ID: 16973711
Correct again, Mr Russell.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
This tutorial demonstrates a quick way of adding group price to multiple Magento products.
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now