Link to home
Start Free TrialLog in
Avatar of kadolika
kadolika

asked on

syslog unix file

I try to analyze a sylog Unix out statistics  such as the number of marks for each service or the number of connections pop.I want do it in Perl.
syslog.txt
ASKER CERTIFIED SOLUTION
Avatar of arnold
arnold
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of kadolika
kadolika

ASKER

Thanks Arnold
for perl script could you provide me a function of the inputs of my syslog.
Syslog has different Sets that are configurable in /etc/syslog.conf or /etc/rsyslog.conf

The pattern match could contain three variants for the date
Jan 17 hh:mm:ss host service/application[pid]: message
Thu, 17 jan 2015 hh:mm:ss host service/application[pid]: message


What/which data points are you interested in, the current date/time, host, application[pid] message
Extracting subsequent data is not a complex issue.
To that I suggest you check whether the current syslog data is reflecting the information you want including dates.


cat syslog.txt | perl perlscript.pl


#!/usr/bin/perl -w

my ($day, $mon, $year, $day_week, $hh, $mm, $ss, $app,$pid, $messae);

while (<STDIN>) {
chomp();

If ( /^([a-z]+)[,]*\s+(\d+)\s+(\d+):(\d+):(\d+)\s+([a-z0-9\.-]+)\s+([a-z]+)\[(\d+)\]: (.*)$/i ) {
              $mon=$1;
              $day=$2;
              $hh=$3;
              $mm=$4;
              $ss=$5;
              $host=$6;
              $app=$7;
              $pid=$8;
              $message=$9;
} # close if
print "what you want to display as a test $variable\n";

} # close while loop

Open in new window


See if that helps.
Do not attempt further matches until after you assign all the prior matched variables as I have, or your subsequent variable will continue other information.
many thanks arnold

can you help me for that  my script  must analyse the syslog unix file and  give the statictics :
 1- the covered period of the log (start-end) by date and hours;
 2- the total number of lines (traces) foreach adress;
 3-the total numbers of traces for each service;
 4-the total number of connections pop, ssh and imap;
 5- the list of addresses that made a ssh connection and how many for each
 6- list of adresses that sent an email by sendmail and how many for each

Can you help me for 2 or 3 points, please.
1, look at the syslog conf file to include the year (%y) if I am not mistaken.

To capture the start use a variable ($firstline initially defined as false equal to 0.) in the first match add the if ( $firstline==0 ) { start="$mon $day $hh:$mm:$ss"; $firstline=1; } #(meaning firstline was found)

You can have counters
Using hashes
$counter for the total lines
$counters{'$app'} +=1; # application line counter
4,5,6 are a subset of counters that require secondary parsing of the meSsage portion of the intitial parsing scheme.
How familiar are you with hash of hashes and has reference/de reference?
Many thanks arnold
I am a beginner to these different types of hashes but it goes with  reference/de reference
my %hash

$hash{'$app'}+=1;
Check if message context is pop
$hash{'$app'}->{'pop'}+=1;
Check if message context is IMAP
$hash{'$app'}->{'imap'}+=1;
Check message, type and ip
$hash{'$app'}->{$type}->{$ip}+=1;

For ssh, the app will count that, tge message you would

To get type you would use outer loop will be the enumeration of apps in the hash.
foreach ($type keys %($hash{'$app'}) {
             foreach ( $ipaddress keys %($hash{'$app'}->{$type}) {
                            here you can print the service/type/ip counter........
              }
}
many thanks arnold
hi Arnold

i don't understand this part can you give me more details please.
Check message, type and ip
$hash{'$app'}->{$type}->{$ip}+=1;

For ssh, the app will count that, tge message you would

To get type you would use outer loop will be the enumeration of apps in the hash.
foreach ($type keys %($hash{'$app'}) {
             foreach ( $ipaddress keys %($hash{'$app'}->{$type}) {
                            here you can print the service/type/ip counter........
              }
}
%hash
$hash{'record'} directly access value
$hash->{'Record'} hash reference
Creating multilevel hash(hash of hashes)
Hash
         Record
                       Sub record
ok thanks Arnold
You could do $hash{'record'}{'sub record'}....
The referencing and maintaining readability ............
Referencing dereferencing are explained if you look at array of arrays, hashes or hash of arrays, hashes.

Hash -> array
% to de reference a hash and @ to dereference an array
thanks arnold
Hi Arnold
please, can i you give an example for the 4the point please,
Sure

If  ( $app=='sendmail' && $message =~ /(pop|imap)/i ) {
    #       $1 will either be pop or imap when the app is sendmail adding data points to extract
#Do stuff
}
thanks arnold
hi Arnold how I get the counter results
foreach will run through the reference hash

Foreach ($variable keys %hash_used) {

R

You would then use the variable as the parameter $hash{$application} to get the general app counter.
You would then use a nested foreach with %($hash{$application}) with $hash{$application}->{$type} has the application/type counter
This will assign the variable with the sub category.
You would need to nest as many foreach levels of hash you have

Outer look (apps)
 1inner loop, types
    2 inner loop, in the event you want/collect information I.e. IP address based counters for the application, type, source,
        3 inner loop that will count success/failure from the IP....
           Etc.