kadolika
asked on
syslog unix file
I try to analyze a sylog Unix out statistics such as the number of marks for each service or the number of connections pop.I want do it in Perl.
syslog.txt
syslog.txt
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Syslog has different Sets that are configurable in /etc/syslog.conf or /etc/rsyslog.conf
The pattern match could contain three variants for the date
Jan 17 hh:mm:ss host service/application[pid]: message
Thu, 17 jan 2015 hh:mm:ss host service/application[pid]: message
What/which data points are you interested in, the current date/time, host, application[pid] message
Extracting subsequent data is not a complex issue.
To that I suggest you check whether the current syslog data is reflecting the information you want including dates.
cat syslog.txt | perl perlscript.pl
See if that helps.
Do not attempt further matches until after you assign all the prior matched variables as I have, or your subsequent variable will continue other information.
The pattern match could contain three variants for the date
Jan 17 hh:mm:ss host service/application[pid]: message
Thu, 17 jan 2015 hh:mm:ss host service/application[pid]: message
What/which data points are you interested in, the current date/time, host, application[pid] message
Extracting subsequent data is not a complex issue.
To that I suggest you check whether the current syslog data is reflecting the information you want including dates.
cat syslog.txt | perl perlscript.pl
#!/usr/bin/perl -w
my ($day, $mon, $year, $day_week, $hh, $mm, $ss, $app,$pid, $messae);
while (<STDIN>) {
chomp();
If ( /^([a-z]+)[,]*\s+(\d+)\s+(\d+):(\d+):(\d+)\s+([a-z0-9\.-]+)\s+([a-z]+)\[(\d+)\]: (.*)$/i ) {
$mon=$1;
$day=$2;
$hh=$3;
$mm=$4;
$ss=$5;
$host=$6;
$app=$7;
$pid=$8;
$message=$9;
} # close if
print "what you want to display as a test $variable\n";
} # close while loop
See if that helps.
Do not attempt further matches until after you assign all the prior matched variables as I have, or your subsequent variable will continue other information.
ASKER
many thanks arnold
can you help me for that my script must analyse the syslog unix file and give the statictics :
1- the covered period of the log (start-end) by date and hours;
2- the total number of lines (traces) foreach adress;
3-the total numbers of traces for each service;
4-the total number of connections pop, ssh and imap;
5- the list of addresses that made a ssh connection and how many for each
6- list of adresses that sent an email by sendmail and how many for each
Can you help me for 2 or 3 points, please.
can you help me for that my script must analyse the syslog unix file and give the statictics :
1- the covered period of the log (start-end) by date and hours;
2- the total number of lines (traces) foreach adress;
3-the total numbers of traces for each service;
4-the total number of connections pop, ssh and imap;
5- the list of addresses that made a ssh connection and how many for each
6- list of adresses that sent an email by sendmail and how many for each
Can you help me for 2 or 3 points, please.
1, look at the syslog conf file to include the year (%y) if I am not mistaken.
To capture the start use a variable ($firstline initially defined as false equal to 0.) in the first match add the if ( $firstline==0 ) { start="$mon $day $hh:$mm:$ss"; $firstline=1; } #(meaning firstline was found)
You can have counters
Using hashes
$counter for the total lines
$counters{'$app'} +=1; # application line counter
4,5,6 are a subset of counters that require secondary parsing of the meSsage portion of the intitial parsing scheme.
How familiar are you with hash of hashes and has reference/de reference?
To capture the start use a variable ($firstline initially defined as false equal to 0.) in the first match add the if ( $firstline==0 ) { start="$mon $day $hh:$mm:$ss"; $firstline=1; } #(meaning firstline was found)
You can have counters
Using hashes
$counter for the total lines
$counters{'$app'} +=1; # application line counter
4,5,6 are a subset of counters that require secondary parsing of the meSsage portion of the intitial parsing scheme.
How familiar are you with hash of hashes and has reference/de reference?
ASKER
Many thanks arnold
ASKER
I am a beginner to these different types of hashes but it goes with reference/de reference
my %hash
$hash{'$app'}+=1;
Check if message context is pop
$hash{'$app'}->{'pop'}+=1;
Check if message context is IMAP
$hash{'$app'}->{'imap'}+=1 ;
Check message, type and ip
$hash{'$app'}->{$type}->{$ ip}+=1;
For ssh, the app will count that, tge message you would
To get type you would use outer loop will be the enumeration of apps in the hash.
foreach ($type keys %($hash{'$app'}) {
foreach ( $ipaddress keys %($hash{'$app'}->{$type}) {
here you can print the service/type/ip counter........
}
}
$hash{'$app'}+=1;
Check if message context is pop
$hash{'$app'}->{'pop'}+=1;
Check if message context is IMAP
$hash{'$app'}->{'imap'}+=1
Check message, type and ip
$hash{'$app'}->{$type}->{$
For ssh, the app will count that, tge message you would
To get type you would use outer loop will be the enumeration of apps in the hash.
foreach ($type keys %($hash{'$app'}) {
foreach ( $ipaddress keys %($hash{'$app'}->{$type}) {
here you can print the service/type/ip counter........
}
}
ASKER
many thanks arnold
ASKER
hi Arnold
i don't understand this part can you give me more details please.
Check message, type and ip
$hash{'$app'}->{$type}->{$ ip}+=1;
For ssh, the app will count that, tge message you would
To get type you would use outer loop will be the enumeration of apps in the hash.
foreach ($type keys %($hash{'$app'}) {
foreach ( $ipaddress keys %($hash{'$app'}->{$type}) {
here you can print the service/type/ip counter........
}
}
i don't understand this part can you give me more details please.
Check message, type and ip
$hash{'$app'}->{$type}->{$
For ssh, the app will count that, tge message you would
To get type you would use outer loop will be the enumeration of apps in the hash.
foreach ($type keys %($hash{'$app'}) {
foreach ( $ipaddress keys %($hash{'$app'}->{$type}) {
here you can print the service/type/ip counter........
}
}
%hash
$hash{'record'} directly access value
$hash->{'Record'} hash reference
Creating multilevel hash(hash of hashes)
Hash
Record
Sub record
$hash{'record'} directly access value
$hash->{'Record'} hash reference
Creating multilevel hash(hash of hashes)
Hash
Record
Sub record
ASKER
ok thanks Arnold
You could do $hash{'record'}{'sub record'}....
The referencing and maintaining readability ............
The referencing and maintaining readability ............
Referencing dereferencing are explained if you look at array of arrays, hashes or hash of arrays, hashes.
Hash -> array
% to de reference a hash and @ to dereference an array
Hash -> array
% to de reference a hash and @ to dereference an array
ASKER
thanks arnold
ASKER
Hi Arnold
please, can i you give an example for the 4the point please,
please, can i you give an example for the 4the point please,
Sure
If ( $app=='sendmail' && $message =~ /(pop|imap)/i ) {
# $1 will either be pop or imap when the app is sendmail adding data points to extract
#Do stuff
}
If ( $app=='sendmail' && $message =~ /(pop|imap)/i ) {
# $1 will either be pop or imap when the app is sendmail adding data points to extract
#Do stuff
}
ASKER
thanks arnold
ASKER
hi Arnold how I get the counter results
foreach will run through the reference hash
Foreach ($variable keys %hash_used) {
R
You would then use the variable as the parameter $hash{$application} to get the general app counter.
You would then use a nested foreach with %($hash{$application}) with $hash{$application}->{$typ e} has the application/type counter
This will assign the variable with the sub category.
You would need to nest as many foreach levels of hash you have
Outer look (apps)
1inner loop, types
2 inner loop, in the event you want/collect information I.e. IP address based counters for the application, type, source,
3 inner loop that will count success/failure from the IP....
Etc.
Foreach ($variable keys %hash_used) {
R
You would then use the variable as the parameter $hash{$application} to get the general app counter.
You would then use a nested foreach with %($hash{$application}) with $hash{$application}->{$typ
This will assign the variable with the sub category.
You would need to nest as many foreach levels of hash you have
Outer look (apps)
1inner loop, types
2 inner loop, in the event you want/collect information I.e. IP address based counters for the application, type, source,
3 inner loop that will count success/failure from the IP....
Etc.
ASKER
for perl script could you provide me a function of the inputs of my syslog.