Fail2ban Stopped Jailing

sharingsunshine
sharingsunshine used Ask the Experts™
on
Fail2ban stopped working.  I have scoured the log files and no errors.  It was working and it seems since my last Yum update for security it quit working.  I can do fail2ban-client status and it shows 7 jails.  I can look in the iptables and it shows the jails.  However, when I run fail2ban-regex it shows many hits but none are getting blocked by the iptables.  The iptables are on.

This is 0.8.14 and I am on a Linux 2 Amazon Ami with PHP 7 and Python 2.7.14.

When it was working  it had over 221 ips banned just in one jail.

Please help me get this going.  The bots overrun my system if it isn't in place.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
David FavorFractional CTO
Distinguished Expert 2018

Commented:
Check to see if fail2ban is running first...

ps auxww | grep fail2ban

Open in new window


If not, likely the following will work...

service fail2ban restart

Open in new window


Then check /var/log/* entries for problems if fail2ban fails to restart.

I normally setup fail2ban to log to it's own log file...

Usually you can change this in fail2ban.conf logtarget=/var/log/fail2ban.log

At least this works on most Distros.

Author

Commented:
it is running -
root     27940 27.2  0.1 1235908 13516 ?       Sl   22:24   1:39 /bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
root     29287  0.0  0.0 119464   924 pts/2    S+   22:30   0:00 grep --color=auto fail2ban

Open in new window


So how do I get it to jail the log hits?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Is the method of the jail changed? and still available?...
if the original fail2ban .conf files  have been updated by you for use then they maybe overwritten by the install preventing some action.
For personal use and activatation of jails use  *.local files not the *.conf files.

in recent fail2ban some changes have beenmade to activations, filters and jails  so old jails might need some adjustments.

Besides that the fail2ban logging should give some indication about what it does.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
nothing has changed related to Fail2Ban.  I remembered my configuration was suspect on the 0.8 distribution.  I think that is where the problem lies.  

I haven't overwritten anything and it was working perfectly and then stopped.  I found out because all of the bot activity crashed my instance.

I have looked into the fail2ban.log and didn't find anything.  If you can tell me what to search for I will give it another go.

Thanks,
David FavorFractional CTO
Distinguished Expert 2018

Commented:
You're correct. Something has changed + now you have to track down what changed.

Follow my instructions above for debugging this.
fail2ban-client -v -f start
should tell you which problems there are (you might need to stop fail2ban-server)

Author

Commented:
I have already stated that I have looked extensively in /var/log*.  If you have a suggestion for a keyword to use that would be very helpful.  I have searched for fail2ban, failed, error.

If you know any command line snippets they would be extremely helpful to know.

Author

Commented:
Bernard,  How do I know which are the problems?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
Can you show a jail snippet and all filters & actions that are mentioned in it?

Author

Commented:
[block-bot-name]
enabled = true
filter = block-bot-name
action   = iptables[name=block-bot-name, port=443, protocol=tcp]
           mail-buffered[name=block-bot-name, dest=ec2-user@localhost, sender=root@abc.com]
logpath = /path-to-log
maxretry = 1
bantime = 31556736
findtime =1

Open in new window


# Fail2Ban configuration file
#
# Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# above mentioned bots.


[Definition]


failregex = ^<HOST> - - .*(pingdom\.com|ahrefs\.com|yandex\.com|dotbot\.com|semrush\.com)



ignoreregex =

Open in new window


[root@ip-172-31-8-214 action.d]# cat iptables.conf
# Fail2Ban configuration file
#
# Author: Cyril Jaquier
#
#

[INCLUDES]

before = iptables-blocktype.conf

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> --dport <port> -j fail2ban-<name>

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = iptables -D <chain> -p <protocol> --dport <port> -j fail2ban-<name>
             iptables -F fail2ban-<name>
             iptables -X fail2ban-<name>

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = iptables -n -L <chain> | grep -q 'fail2ban-<name>[ \t]'

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j <blocktype>

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = iptables -D fail2ban-<name> -s <ip> -j <blocktype>

[Init]

# Default name of the chain
#
name = default

# Option:  port
# Notes.:  specifies port to monitor
# Values:  [ NUM | STRING ]  Default:
#
port = ssh

# Option:  protocol
# Notes.:  internally used by config reader for interpolations.
# Values:  [ tcp | udp | icmp | all ] Default: tcp
#
protocol = tcp

# Option:  chain
# Notes    specifies the iptables chain to which the fail2ban rules should be
#          added
# Values:  STRING  Default: INPUT
chain = INPUT

Open in new window


[root@ip-172-31-8-214 ~]# fail2ban-regex /var/log/httpd/access_log /etc/fail2ban/filter.d/block-bot-name.conf

Running tests
=============

Use   failregex file : /etc/fail2ban/filter.d/block-bot-name.conf
Use         log file : /var/log/httpd/access_log


Results
=======

Failregex: 231 total
|-  #) [# of hits] regular expression
|   1) [231] ^<HOST> - - .*(pingdom\.com|ahrefs\.com|yandex\.com|dotbot\.com|semrush\.com)
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [62327] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 62327 lines, 0 ignored, 231 matched, 62096 missed
Missed line(s): too many to print.  Use --print-all-missed to print all 62096 lines

Open in new window


I don't have that many matches in my iptables

Chain fail2ban-block-bot-name (1 references)
num  target     prot opt source               destination
1    REJECT     all  --  100-43-91-119.spider.yandex.com  anywhere             reject-with icmp-port-unreachable
2    RETURN     all  --  anywhere             anywhere

Open in new window


Looking at the log file you can see another yandex bot came by too after I restarted fail2ban at 8:57 AM.

[root@ip-172-31-8-214 filter.d]# tail -100 /var/log/httpd/access_log | grep yandex
100.43.91.101 - - [29/Aug/2018:09:09:06 -0400] "GET /wp-content/uploads/2013/12/gooddog.gif HTTP/1.1" 200 4994 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
[root@ip-172-31-8-214 filter.d]# tail -500 /var/log/httpd/access_log | grep yandex
100.43.91.101 - - [29/Aug/2018:08:43:45 -0400] "GET /877-large_default/VS_C_TCM_Concentrate_Chinese_p_721.jpg HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
100.43.91.119 - - [29/Aug/2018:08:59:30 -0400] "GET /robots.txt HTTP/1.1" 200 99 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
100.43.91.101 - - [29/Aug/2018:09:09:06 -0400] "GET /wp-content/uploads/2013/12/gooddog.gif HTTP/1.1" 200 4994 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
[
[

Open in new window

nociSoftware Engineer
Distinguished Expert 2018

Commented:
Looks good....
are the fail2ban-* chains still referenced from INPUT?

iptables --line-numbers -nv -L INPUT

Open in new window

Author

Commented:
[root@ip-172-31-8-214 fail2ban]# iptables --line-numbers -nv -L INPUT
Chain INPUT (policy ACCEPT 5598 packets, 559K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    10242 1109K fail2ban-block-bot-name  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
2    10235 1109K fail2ban-speed-move  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
3    10235 1109K fail2ban-login-multi  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
4    10235 1109K fail2ban-wordpress-soft  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
5    10133 1092K fail2ban-wordpress-hard  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
6    10133 1092K fail2ban-headless  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
7    12379 1307K fail2ban-BadBots  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443
8    16279  920K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set rukpuabucnro-block src
9     1776  103K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set ngtuczbrafal-block src

Open in new window

nociSoftware Engineer
Distinguished Expert 2018

Commented:
So those are hit 10K+ times.
 I don't see the hitcounts in the earlier examples:

what does the following show?

iptables --line-numbers -nv -L fail2ban-block-bot-name
iptables --line-numbers -nv -L fail2ban-speed-move
iptables --line-numbers -nv -L fail2ban-BadBots

Open in new window


Looking again at your rules, the variouslog  lines need to have a short time between them... to count.
(1 second..., specify 1h if you want a match to count with another one in 1hour, that requires a recent fail2ban)
it is about findtime, make that either 3600 or a much longer time.

I am running 0.10.3 btw.

Author

Commented:
[root@ip-172-31-8-214 ~]# iptables --line-numbers -nv -L fail2ban-block-bot-name
Chain fail2ban-block-bot-name (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 REJECT     all  --  *      *       46.229.168.69        0.0.0.0/0            reject-with icmp-port-unreachable
2        0     0 REJECT     all  --  *      *       46.229.168.71        0.0.0.0/0            reject-with icmp-port-unreachable
3       20  1148 REJECT     all  --  *      *       100.43.91.124        0.0.0.0/0            reject-with icmp-port-unreachable
4       20  1148 REJECT     all  --  *      *       100.43.90.103        0.0.0.0/0            reject-with icmp-port-unreachable
5       20  1148 REJECT     all  --  *      *       100.43.81.100        0.0.0.0/0            reject-with icmp-port-unreachable
6       20  1148 REJECT     all  --  *      *       100.43.85.167        0.0.0.0/0            reject-with icmp-port-unreachable
7       25  1456 REJECT     all  --  *      *       100.43.81.104        0.0.0.0/0            reject-with icmp-port-unreachable
8       25  1448 REJECT     all  --  *      *       100.43.85.107        0.0.0.0/0            reject-with icmp-port-unreachable
9       22  1268 REJECT     all  --  *      *       100.43.90.117        0.0.0.0/0            reject-with icmp-port-unreachable
10      20  1152 REJECT     all  --  *      *       100.43.81.129        0.0.0.0/0            reject-with icmp-port-unreachable
11      20  1144 REJECT     all  --  *      *       100.43.85.108        0.0.0.0/0            reject-with icmp-port-unreachable
12       0     0 REJECT     all  --  *      *       46.229.168.74        0.0.0.0/0            reject-with icmp-port-unreachable
13       0     0 REJECT     all  --  *      *       46.229.168.79        0.0.0.0/0            reject-with icmp-port-unreachable
14       2   120 REJECT     all  --  *      *       46.229.168.77        0.0.0.0/0            reject-with icmp-port-unreachable
15      26  1512 REJECT     all  --  *      *       100.43.91.113        0.0.0.0/0            reject-with icmp-port-unreachable
16      23  1328 REJECT     all  --  *      *       100.43.91.105        0.0.0.0/0            reject-with icmp-port-unreachable
17       0     0 REJECT     all  --  *      *       46.229.168.73        0.0.0.0/0            reject-with icmp-port-unreachable
18      27  1576 REJECT     all  --  *      *       100.43.91.119        0.0.0.0/0            reject-with icmp-port-unreachable
19      30  1748 REJECT     all  --  *      *       100.43.91.101        0.0.0.0/0            reject-with icmp-port-unreachable
20       2   120 REJECT     all  --  *      *       46.229.168.78        0.0.0.0/0            reject-with icmp-port-unreachable
21       2   120 REJECT     all  --  *      *       46.229.168.65        0.0.0.0/0            reject-with icmp-port-unreachable
22       4   240 REJECT     all  --  *      *       46.229.168.81        0.0.0.0/0            reject-with icmp-port-unreachable
23       4   240 REJECT     all  --  *      *       46.229.168.76        0.0.0.0/0            reject-with icmp-port-unreachable
24       0     0 REJECT     all  --  *      *       46.229.168.84        0.0.0.0/0            reject-with icmp-port-unreachable
25      41  2460 REJECT     all  --  *      *       216.244.66.233       0.0.0.0/0            reject-with icmp-port-unreachable
26       2   120 REJECT     all  --  *      *       46.229.168.75        0.0.0.0/0            reject-with icmp-port-unreachable
27    289K   31M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
[root@ip-172-31-8-214 ~]#

Open in new window


Chain fail2ban-speed-move (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     289K   31M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
[root@ip-172-31-8-214 ~]#

Open in new window


[root@ip-172-31-8-214 ~]# iptables --line-numbers -nv -L fail2ban-BadBots
Chain fail2ban-BadBots (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 REJECT     all  --  *      *       185.24.233.19        0.0.0.0/0            reject-with icmp-port-unreachable
2     353K   37M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Open in new window

Software Engineer
Distinguished Expert 2018
Commented:
So some rejects are hit multiple times....,  and sometimes a remote system only probes once or twice.
(And there are mutiple rejects in rules....).
If you meant  no new ones are added, do adjust the find time, your jail only seems to trigger when filters match multiple accesses within one second.

Author

Commented:
Your request to see my jails caused me to find a syntax error in the filter.  After making the change it began to block several ip's.  I still am having a problem with other jails that used to catch quite frequently all the marauders.  I can only conclude I need to upgrade 0.8.14 to 0.10.3.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial