Link to home
Start Free TrialLog in
Avatar of FireBall
FireBall

asked on

Perl output collect

I have a perl script which generate outputs as this :

1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             ce125b8f1c37b43f2f192850134c38e9c2a46fb81988657188b06fadff92
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             c40be80cc61779ba9c547470efa15dc501f0d20998386ebde1c74cef82d1
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             187edb519a7827391ee2acd9bc84e68c9b844a675f3a779991a5c4b0e0bb
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             3fa4cdd26615f3a65cd6499087776da26899e258fd1c318daa7f550235b2
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    360             34298c65c873740b04d1e670570241db6e02445bbeb6fb2c39c3d3867fb0
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             6d2937f95c2257e831f9731230c8e08a6c08281871a66fa67b2f8572eaef
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             950ae7e2a8b776e0bcc6a4990ff8396b64a5bb186a77529d9dbc57f6cd8d
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             11ac0ece9d32f2df58b2746b89422e980a6fb96793d2f55a06d63c946460
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             a87c443328f08bae55a405129927c73147eff9358666defd5b53dca9995d
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             be9cb3bde0842f9d793c9ee856a211192196d4ba1c93fecf5285239c3f34
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             7c59d3bb055f037733b1bccf309e715e8b89a30c1b68917b0c6268953d12
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             7fa2a32ec4e205b23ecb00bed441d88fe2ee36d7268f3acd7e5bcdb251fe
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             bd106e6077b6188c4270655f461a97b93c2d54837f3022aaca0b7d9083a5
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             bdf3fe96db36c3b0bdad4f30763ab20173ada5263ff380930eb0fd811500
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             dd505f9a0d5ed1b4f29a4d90771c31bba8ca86e32b523041f4ffcd5fc0fa
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             c265f17eebea4f13a7b4be5563fd8882b33f7391193c98b053e197a1e859
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             0acc383dac9a06e675145bff92b16c45a46571811cd1c9a78887a1381c31
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             f43873058321e34b3565be9588fa23cf9c4725502f224713ed80e479d94d
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             c1084085c406087514ace765ffd42d64be41315dd71b4411e9229d064062
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             74957b588e9202c48e593947dcab0e10b6a1f662223d4f4acc4798bb2fa4
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             e1ed2765d45596df8709cdf05717320e5fca81b4a3d228705c2f7b4337ef
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             c654af275921aeca7e5768e59164112497fe512bb3139d4f320cabc75389
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             5dc288d103b831d532599c7b8de6296f20c9e0018c06cce7629c51768e70
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    232             fd2dd49babe7e16178f41b2681baae9387f6d6015b729aa291c35a3c167a
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             7654993bc91ec89e9c03b75058bfb3976ab5fb0f31e03c9026f27922203e
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             2d195bd485e7d0a8760b15b84628b67dce521a9bd005815c87e10cc47f95
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             f5ccc267d66dafcdca144ad060dc4c35fd6e62f17939ab3d86dc57133d53
1439992295000   TCP     16      6       120     78.186.179.127  7609    185.9.159.244   22      40
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    296             f547cdbdafda615dec8728211a6dba030da3e2b84f323f714a14c110198e
1439992295000   TCP     24      6       120     78.186.179.127  7609    185.9.159.244   22      104             140560d1d0c567ef45f16e50f9a2e045c1650b43034c8cb21b27408714f5
1439992295000   TCP     24      6       64      185.9.159.244   22      78.186.179.127  7609    360             2fa7d2c632732993d2bc1f0f4d7987553f3655782c0f0fb0acbe1691c6ab

Open in new window



order of the columns are :
Time
Protocol Name
Flags
Protocol ID
TTL
SRC IP
SRC Port
Dest IP
Dest Port
Data Length
First 30 Byte of data




And i want to find top talkers

Top source IP  got xxxx  query
Top Destination IP  xxxx  query

how should i do it ?
Avatar of wilcoxon
wilcoxon
Flag of United States of America image

This should do it...
use strict;
use warnings;
use Fatal qw(open close);
my $fil = shift or die "Usage: $0 file_to_parse\n";
open IN, $fil or die "could not open $fil: $!";
my (%src, %dest);
while (<IN>) {
    my @f = split;
    $src{$f[5]}++;
    $dest{$f[7]}++;
}
close IN;
my ($max, $maxip) = (0,0);
foreach my $ip (keys %src) {
    if ($src{$ip} > $max) {
        $max = $src{$ip};
        $maxip = $ip;
    }
}
print "Top source IP $maxip got $max query\n";
($max, $maxip) = (0,0);
foreach my $ip (keys %dest) {
    if ($dest{$ip} > $max) {
        $max = $dest{$ip};
        $maxip = $ip;
    }
}
print "Top dest IP $maxip got $max query\n";

Open in new window

Avatar of FireBall
FireBall

ASKER

Actually that is my script :


  #!/usr/bin/perl -w
  use Data::Table;
  use strict;
  use Net::PcapUtils;
  use NetPacket::Ethernet qw(:strip);
  use NetPacket::IP;
  use NetPacket::UDP;
  use NetPacket::TCP;
  use Data::HexDump;
  use Time::HiRes qw( usleep ualarm gettimeofday tv_interval nanosleep
                      clock_gettime clock_getres clock_nanosleep clock
                      stat lstat );



  sub process_pkt {


my $seconds = gettimeofday();
my $ms      = int($seconds*1000);
($seconds, my $fraction) = gettimeofday();
$ms = int(time*1000);



      my($arg, $hdr, $pkt) = @_;
      my $ip_obj = NetPacket::IP->decode(eth_strip($pkt));

my $proto = "$ip_obj->{proto}" ;


if($proto == 17){
      my $udp_obj = NetPacket::UDP->decode($ip_obj->{data});

      print("$ms\t"."UDP\t"."\t"."$ip_obj->{proto}\t"."$ip_obj->{ttl}\t"."$ip_obj->{src_ip}\t"."$udp_obj->{src_port}\t"."$ip_obj->{dest_ip}\t"."$udp_obj->{dest_port}\t"."$ip_obj->{len}\t"."\t");
        my $k=length($udp_obj->{data});
        if ($k>30 ) {$k=30;}
        for (my $i=0;$i<$k;$i++)
        {
                my $hexval = unpack('H2',substr($udp_obj->{data},$i,1));
                print  $hexval;
        }
}
elsif($proto == 6){
      my $tcp_obj = NetPacket::TCP->decode($ip_obj->{data});
      my $tcp_data = NetPacket::TCP::strip($tcp_obj);
      print("$ms\t"."TCP\t"."$tcp_obj->{flags}\t"."$ip_obj->{proto}\t"."$ip_obj->{ttl}\t"."$ip_obj->{src_ip}\t"."$tcp_obj->{src_port}\t"."$ip_obj->{dest_ip}\t"."$tcp_obj->{dest_port}\t"."$ip_obj->{len}\t"."\t");
        my $k=length($tcp_obj->{data});
        if ($k>30 ) {$k=30;}
        for (my $i=0;$i<$k;$i++)
        {
                my $hexval = unpack('H2',substr($tcp_obj->{data},$i,1));
                print  $hexval;
        }


}else{
      my $tcp_obj = NetPacket::TCP->decode($ip_obj->{data});
      my $tcp_data = NetPacket::TCP::strip($tcp_obj);
#      print("$ms\t"."Diger\t"."$tcp_obj->{flags}\t"."$ip_obj->{proto}\t"."$ip_obj->{ttl}\t"."$ip_obj->{src_ip}\t"."$tcp_obj->{src_port}\t"."$ip_obj->{dest_ip}\t"."$tcp_obj->{dest_port}\t"."$ip_obj->{len}\t"."\t");

        my $k=length($tcp_obj->{data});
        if ($k>30 ) {$k=30;}
        for (my $i=0;$i<$k;$i++)
        {
                my $hexval = unpack('H2',substr($tcp_obj->{data},$i,1));
                print  $hexval;
        }


}


print "\n";

}

Net::PcapUtils::loop(\&process_pkt);

Open in new window





And what i want to do is in every one second try to find anomalities.

I will record into some place ip behaviours like

xxx ip gets xxx connection per second if it change more then x times in last one second it will collect data and write an iptables rule from the similar parts like

%80 data same
%80 ttl same
%80 source port same

then it will block on ip tables depending on this values
or simply that should resolve the issu

every unique ip wheter in source or destination got how many connections

like that :
ipaddress      InboundConnectionCount       OutBoundConnectionCount  



is that possible ?
If I'm reading the code correctly, it just loops over process_pkt as often as something new happens, correct?  If so, this should be close to what you want.
#!/usr/bin/perl
use Data::Table;
use strict;
use warnings;
use Net::PcapUtils;
use NetPacket::Ethernet qw(:strip);
use NetPacket::IP;
use NetPacket::UDP;
use NetPacket::TCP;
use Data::HexDump;
use Time::HiRes qw( usleep ualarm gettimeofday tv_interval nanosleep
                    clock_gettime clock_getres clock_nanosleep clock
                    stat lstat );

my %cache;
my $ctime = 0;

sub process_pkt {

    my $seconds = gettimeofday();
    my $ms      = int($seconds*1000);
    ($seconds, my $fraction) = gettimeofday();
    $ms = int(time*1000);

    my($arg, $hdr, $pkt) = @_;
    my $ip_obj = NetPacket::IP->decode(eth_strip($pkt));

    my $proto = "$ip_obj->{proto}" ;

    # if you want it on a different scale than $ms then it's easy to do
    if ($ms > $ctime) {
        foreach my $ip (keys %cache) {
            print "$ip\t$cache{$ip}{src}\t$cache{$ip}{dst}\n";
            # if you want to do more than print out stuff then here is
            # probably best to do it
        }
        $ctime = $ms;
        %cache = ();
    }

    $cache{$ip_obj->{src_ip}}{src}++;
    $cache{$ip_obj->{dest_ip}}{dst}++;

    if($proto == 17){
        my $udp_obj = NetPacket::UDP->decode($ip_obj->{data});

        print("$ms\t"."UDP\t"."\t"."$ip_obj->{proto}\t"."$ip_obj->{ttl}\t"."$ip_obj->{src_ip}\t"."$udp_obj->{src_port}\t"."$ip_obj->{dest_ip}\t"."$udp_obj->{dest_port}\t"."$ip_obj->{len}\t"."\t");
        my $k=length($udp_obj->{data});
        $k = 30 if ($k>30);
        for (my $i=0;$i<$k;$i++) {
            my $hexval = unpack('H2',substr($udp_obj->{data},$i,1));
            print  $hexval;
        }
    }
    elsif($proto == 6){
        my $tcp_obj = NetPacket::TCP->decode($ip_obj->{data});
        my $tcp_data = NetPacket::TCP::strip($tcp_obj);
        print("$ms\t"."TCP\t"."$tcp_obj->{flags}\t"."$ip_obj->{proto}\t"."$ip_obj->{ttl}\t"."$ip_obj->{src_ip}\t"."$tcp_obj->{src_port}\t"."$ip_obj->{dest_ip}\t"."$tcp_obj->{dest_port}\t"."$ip_obj->{len}\t"."\t");
        my $k=length($tcp_obj->{data});
        $k = 30 if ($k>30);
        for (my $i=0;$i<$k;$i++) {
            my $hexval = unpack('H2',substr($tcp_obj->{data},$i,1));
            print  $hexval;
        }
    } else {
        my $tcp_obj = NetPacket::TCP->decode($ip_obj->{data});
        my $tcp_data = NetPacket::TCP::strip($tcp_obj);
#        print("$ms\t"."Diger\t"."$tcp_obj->{flags}\t"."$ip_obj->{proto}\t"."$ip_obj->{ttl}\t"."$ip_obj->{src_ip}\t"."$tcp_obj->{src_port}\t"."$ip_obj->{dest_ip}\t"."$tcp_obj->{dest_port}\t"."$ip_obj->{len}\t"."\t");

        my $k=length($tcp_obj->{data});
        $k = 30 if ($k>30);
        for (my $i=0;$i<$k;$i++) {
            my $hexval = unpack('H2',substr($tcp_obj->{data},$i,1));
            print  $hexval;
        }
    }

    print "\n";
}

Net::PcapUtils::loop(\&process_pkt);

Open in new window

it returns an error for here :

                print "$ip\t$cache{$ip}{src}\t$cache{$ip}{dst}\n";


Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
96.213.254.202          1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
178.20.225.33   1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
228.209.0.0             1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
98.1.0.0                1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
103.33.0.0              3
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
37.123.101.129  1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
227.65.0.0              1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
101.193.0.0             1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
224.0.0.5               18
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
178.20.230.46   1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
3.1.0.0         1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
37.123.98.17    1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
213.238.170.255         2
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
37.123.99.161   1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
99.225.0.0              2
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
100.193.0.0             2
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
185.9.158.158   3
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
178.20.227.129  1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
37.123.100.113  1
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
100.1.0.0               4
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
149.193.37.123  20
Use of uninitialized value in concatenation (.) or string at reader.pl line 97.
1.1.0.0         1

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of wilcoxon
wilcoxon
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial