hard perl script

we are senind the given output below to a subroutine in evey one second.
Than we want to create automated ip table rules with checking thesses
if a destination (side of 255.255.255.255) is in more then 1000+ lines then we want to let the software decide

* if the source port (3.rd column) is a criteria
* if data (4th column) is a criteria
* if destination port is a criteria
* AND TTL check (If attack comes from same machine there must be one ttl value but if it comes from 2 machines ?)


then it will create iptables rules as this

iptables -A input -d (destination ip) -m ttl --ttl-eq=TTLNumber (if ttl is a criteria) --sport xxxx (if source port criteria)  --dport yyyy (if destination port criteria)  --match string --algo kmp --hex-string '|f4 6d 04 25 b2 02 00 0a|'  (if packet is a criteria)  -j DROP


how should we do that ?


1433301992474 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301992503 - 41.26.121.86.   51594   0001000000000000067178737a6b6603        37.123.100.8.   53      233     66
1433301992532 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301992565 - 185.9.156.26.   62218   4e495431006500008802fd66d3000000        185.9.156.13.   9987    128     62
1433301992594 - 185.9.156.26.   62218   670afe6f00000000229d748b45aa7bef        185.9.156.13.   9987    128     212
1433301992640 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301992673 - 213.238.170.199.        137     000100000000000020454a4543454e45        213.238.170.255.        137     128     78
1433301992706 - 213.238.170.199.        137     00010000000000002045434641454d46        213.238.170.255.        137     128     78
1433301992736 - 213.238.170.199.        137     00010000000000002045474545464846        213.238.170.255.        137     128     78
1433301992764 - 213.238.170.199.        137     00010000000000002045424549454e45        213.238.170.255.        137     128     78
1433301682696 - 213.238.170.199.        137     0001000000000000204649464a454b45        213.238.170.255.        137     128     78
1433301682746 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301682778 - 213.238.170.199.        137     00010000000000002046424550454946        213.238.170.255.        137     128     78
1433301682808 - 157.56.205.8.   40021   00010000000000000a636961686f7374        37.123.100.220. 53      112     60
1433301682837 - 191.234.67.56.  44197   00010000000000000a636961686f7374        37.123.100.220. 53      48      60
1433301682866 - 157.56.205.9.   29205   00010000000000000a636961686f7374        37.123.100.220. 53      112     60
1433301682895 - 37.123.103.4.   64210   00010000000000000477706164000001        224.0.0.252.    5355    1       50
1433301682925 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301682954 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301682984 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683012 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683066 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683109 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683139 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683173 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683202 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683232 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683261 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683308 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683340 - 57.177.212.60.  17824   0001000000000000026d740737373737        37.123.100.8.   53      239     60
1433301683370 - 37.123.103.4.   63257   00010000000000000477706164000001        224.0.0.252.    5355    1       50
1433301683399 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683437 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683480 - 157.56.205.9.   14975   00010000000000000a636961686f7374        37.123.100.220. 53      111     60
1433301683510 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683540 - 37.123.103.4.   63257   00010000000000000477706164000001        224.0.0.252.    5355    1       50
1433301683569 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683598 - 37.123.96.154.  137     00010000000000002046484641454245        37.123.96.159.  137     128     78
1433301683632 - 18.92.167.121.  51761   00010000000000000273720377777705        37.123.100.8.   53      233     62
1433301683662 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683691 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683720 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683749 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683778 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683807 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683836 - 061632. 246284  000000000100020222022300672c52  (hlim   UDP     00      128
1433301683869 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301683899 - 37.123.103.4.   137     00010000000000002046484641454245        37.123.103.15.  137     128     78
1433301482893 - 185.9.159.186.  137     00010000000000002045504644454e45        185.9.159.255.  137     128     78
1433301482920 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301482934 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301482947 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301482960 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301482984 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301482997 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483010 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483023 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483058 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483073 - 14.211.48.149.  19515   00010000000000000a656e6978736a69        37.123.100.8.   53      237     68
1433301483102 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483115 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483127 - 185.9.156.26.   62218   4e495431006500008802fd66d3000000        185.9.156.13.   9987    128     62
1433301483140 - 185.9.156.26.   62218   670afe6f00000000229d748b45aa7bef        185.9.156.13.   9987    128     212
1433301483153 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483166 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483178 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483191 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483213 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483228 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483241 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483254 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483266 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483279 - 91.19.112.252.  9508    00010000000000001067627364736467        37.123.100.8.   53      236     77
1433301483291 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483304 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483316 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483329 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483341 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483354 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483366 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483379 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483391 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483404 - 37.123.103.4.   137     00010000000000002046484641454245        37.123.103.15.  137     128     78
1433301483416 - 178.20.225.235. 54513   0500000000000000f8be000013000000        255.255.255.255.        48887   128     75
1433301483433 - 93.158.178.83.  21776   0001000000000000037777770c726f6d        37.123.100.2.   53      51      66

Open in new window

FireBallITAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ozoCommented:
I'm not sure I understand what you are asking.
What is a "destination"?  What is a "side of 255.255.255.255"?  
What should the software decide?
What is a criteria ?
0
FireBallITAuthor Commented:
Destination mean destination ip address
255.255.255.255 mean in the given logs first line you will see it. I tryed to describe that column

If destination has 1000+ log in sended log to sub function
Sub function will check and detect how to block this attack
0
ozoCommented:
So check for 1000+ repeated values in column 6, and find a set of other column values common to those rows?
Wouldn't the destination always be a common value?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

FireBallITAuthor Commented:
Depending on attack structure
Packet should be unique or maybe not
Source should be spood  or maybe not
Ttl must be one if attack come from a source but if it comes from 2 different source ? So we need group the destination with ttl
Source port should be uniqe or maybe not
packet size should be unique or maybe not

Most prof attacks make everything unique so the only criteria is ttl and dest. Ip

But that should block too many people , but what if hacker lost a point like same packets or like packetsize ....etc we will add it to rule
0
ozoCommented:
Which column is Packet? which column is Source? What does a spood look like?  If Destination is unique, how can it be 1000+?
If we only block unique attacks, does that mean that a repeated attack will never get blocked?
0
FireBallITAuthor Commented:
Screenshot-18.png
0
ozoCommented:
Or if attacks are never repeated, then why bother blocking anything we've seen before?
0
FireBallITAuthor Commented:
of course if server gets 1500 logs 1500 of them should not be an attack . 800 of them maybe attackler log
So after grouping by TTL and destination it will check for the density of other columns same values for ex. our limit is %80 for aaccepting as an attack (because %20 should be real traffic from the same ttl)

IT will be detect that
from same ttl and to same dest. software found that %80 packet sample is same %2 source port same %1 source ip same %90 packet size same ....etc.

so if it is more then %80 will be a criteria for created iptables rule
0
FireBallITAuthor Commented:
That scripts loop in one second so after attack start it blocks max. in 1 second and release back the rule in 5 minutes.
instead of manuelly checking that would be perfecct.
we have some enterprise products like juniper srx 3k , 5k , mx480 ...etc.
Any of this systems are not valuable on udp attacks because they work in session logic and can not decide spoof udp so we will bring a transparent bridge between router and firewall which will clean up the dirty traffic
and we are always under attack %98 of this attacks should be catched by this filter
If this attack hits and goes in to firewall it locks all the system
If we limit the firewall udp threshold it blocks real connections this is a type of clean up work of traffic
0
ozoCommented:
0
FireBallITAuthor Commented:
If a server sends packets to a destination TTL never changes.
because TTL was count depending on the router count on the road.
so If attack comes from a server TTL never changes , If attack comes from 2 servers there should be 2 different TTL value in the density of the logs depending on normal traffic vs attack traffick

so TTL & destination grouped  ip log count shows us the attacker's traffic probably.
Of course there should be real customers from the same ttl other criteria's allow us to block attack specificly and less customer to be effected
0
ozoCommented:
#!/bin/perl
use strict;
use warnings;
my %log;
while( <> ){
    my @F=split;
    push @{$log{"@F[5,7]"}},[@F];  # group by destination, TTL
}
for( values %log ){
    next if @$_ < 1000;  # check for 1000+ repeated values
    my %count;
    print @$_." @{$_->[0]}[5,7]\n";
    for( @$_ ){
      for my $i ( 0..4,6,8..$#$_ ){
      ++$count{$i,$_->[$i]}; # check for the density of other columns same values
        }
    }
    for( sort{$count{$b}<=>$count{$a} || $a cmp $b} keys %count ){
        print s/$;/ /r, " $count{$_}\n";
      last if $count{$_} < 800; # limit is %80
   }
}
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
FireBallITAuthor Commented:
thank you for this good response ;

 print("$ms"."\t"."$ip_obj->{ttl}"."\t"."$ip_obj->{src_ip}"."\t"."$udp_obj->{src_port}"."\t",
                   "$ip_obj->{dest_ip}"."\t"."$udp_obj->{dest_port}"."\t","$udp_obj->{len}"."\t"."$fragment \n");


we got the values as this not from the screen your code seems ok but i do not know how to compare them :)
0
FireBallITAuthor Commented:
We have made some changes and resolved some bugs.
This is working better but could not decide how to solve the flow descisions

Screenshot-19.png

1433361008586   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361035972   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361059760   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361065737   128     37.123.102.245  137     37.123.102.255  137     58      bc1b0110000100000000000020464944
1433361083986   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361092581   1       37.123.103.30   61087   224.0.0.252     5355    30      326d00000001000000000000047770
1433361108382   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361131553   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361155009   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361175924   128     37.123.101.67   137     37.123.101.255  137     58      e83d0110000100000000000020464846
1433361179300   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361193085   1       37.123.103.30   61087   224.0.0.252     5355    30      326d00000001000000000000047770
1433361203707   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361228626   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361251772   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361276255   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361300329   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361323604   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361348029   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361371867   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361396217   128     178.20.225.235  54513   255.255.255.255 48887   55      2f0000000500000000000000f8be0000
1433361396326   128     37.123.103.30   137     37.123.103.31   137     58      de690110000100000000000020464846

Open in new window

0
FireBallITAuthor Commented:
also we have faced with another problem - sampling rate-
tcpdump or net::pcap packages
returns the results in rates not a real time flow.
So it can not catch the 1 second easily. some times if there is no heavy traffic it returns the lines in 3 seconds that should makes every thing complicated :(
maybe we need to use mysql for operations with recording datas
0
FireBallITAuthor Commented:
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Perl

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.