Fail2ban Stopped Jailing

Fail2ban stopped working.  I have scoured the log files and no errors.  It was working and it seems since my last Yum update for security it quit working.  I can do fail2ban-client status and it shows 7 jails.  I can look in the iptables and it shows the jails.  However, when I run fail2ban-regex it shows many hits but none are getting blocked by the iptables.  The iptables are on.

This is 0.8.14 and I am on a Linux 2 Amazon Ami with PHP 7 and Python 2.7.14.

When it was working  it had over 221 ips banned just in one jail.

Please help me get this going.  The bots overrun my system if it isn't in place.
sharingsunshineAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Check to see if fail2ban is running first...

ps auxww | grep fail2ban

Open in new window


If not, likely the following will work...

service fail2ban restart

Open in new window


Then check /var/log/* entries for problems if fail2ban fails to restart.

I normally setup fail2ban to log to it's own log file...

Usually you can change this in fail2ban.conf logtarget=/var/log/fail2ban.log

At least this works on most Distros.
0
sharingsunshineAuthor Commented:
it is running -
root     27940 27.2  0.1 1235908 13516 ?       Sl   22:24   1:39 /bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
root     29287  0.0  0.0 119464   924 pts/2    S+   22:30   0:00 grep --color=auto fail2ban

Open in new window


So how do I get it to jail the log hits?
0
nociSoftware EngineerCommented:
Is the method of the jail changed? and still available?...
if the original fail2ban .conf files  have been updated by you for use then they maybe overwritten by the install preventing some action.
For personal use and activatation of jails use  *.local files not the *.conf files.

in recent fail2ban some changes have beenmade to activations, filters and jails  so old jails might need some adjustments.

Besides that the fail2ban logging should give some indication about what it does.
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

sharingsunshineAuthor Commented:
nothing has changed related to Fail2Ban.  I remembered my configuration was suspect on the 0.8 distribution.  I think that is where the problem lies.  

I haven't overwritten anything and it was working perfectly and then stopped.  I found out because all of the bot activity crashed my instance.

I have looked into the fail2ban.log and didn't find anything.  If you can tell me what to search for I will give it another go.

Thanks,
0
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
You're correct. Something has changed + now you have to track down what changed.

Follow my instructions above for debugging this.
0
Bernard S.CTOCommented:
fail2ban-client -v -f start
should tell you which problems there are (you might need to stop fail2ban-server)
0
sharingsunshineAuthor Commented:
I have already stated that I have looked extensively in /var/log*.  If you have a suggestion for a keyword to use that would be very helpful.  I have searched for fail2ban, failed, error.

If you know any command line snippets they would be extremely helpful to know.
0
sharingsunshineAuthor Commented:
Bernard,  How do I know which are the problems?
0
nociSoftware EngineerCommented:
Can you show a jail snippet and all filters & actions that are mentioned in it?
0
sharingsunshineAuthor Commented:
[block-bot-name]
enabled = true
filter = block-bot-name
action   = iptables[name=block-bot-name, port=443, protocol=tcp]
           mail-buffered[name=block-bot-name, dest=ec2-user@localhost, sender=root@abc.com]
logpath = /path-to-log
maxretry = 1
bantime = 31556736
findtime =1

Open in new window


# Fail2Ban configuration file
#
# Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# above mentioned bots.


[Definition]


failregex = ^<HOST> - - .*(pingdom\.com|ahrefs\.com|yandex\.com|dotbot\.com|semrush\.com)



ignoreregex =

Open in new window


[root@ip-172-31-8-214 action.d]# cat iptables.conf
# Fail2Ban configuration file
#
# Author: Cyril Jaquier
#
#

[INCLUDES]

before = iptables-blocktype.conf

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> --dport <port> -j fail2ban-<name>

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = iptables -D <chain> -p <protocol> --dport <port> -j fail2ban-<name>
             iptables -F fail2ban-<name>
             iptables -X fail2ban-<name>

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = iptables -n -L <chain> | grep -q 'fail2ban-<name>[ \t]'

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j <blocktype>

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = iptables -D fail2ban-<name> -s <ip> -j <blocktype>

[Init]

# Default name of the chain
#
name = default

# Option:  port
# Notes.:  specifies port to monitor
# Values:  [ NUM | STRING ]  Default:
#
port = ssh

# Option:  protocol
# Notes.:  internally used by config reader for interpolations.
# Values:  [ tcp | udp | icmp | all ] Default: tcp
#
protocol = tcp

# Option:  chain
# Notes    specifies the iptables chain to which the fail2ban rules should be
#          added
# Values:  STRING  Default: INPUT
chain = INPUT

Open in new window


[root@ip-172-31-8-214 ~]# fail2ban-regex /var/log/httpd/access_log /etc/fail2ban/filter.d/block-bot-name.conf

Running tests
=============

Use   failregex file : /etc/fail2ban/filter.d/block-bot-name.conf
Use         log file : /var/log/httpd/access_log


Results
=======

Failregex: 231 total
|-  #) [# of hits] regular expression
|   1) [231] ^<HOST> - - .*(pingdom\.com|ahrefs\.com|yandex\.com|dotbot\.com|semrush\.com)
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [62327] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 62327 lines, 0 ignored, 231 matched, 62096 missed
Missed line(s): too many to print.  Use --print-all-missed to print all 62096 lines

Open in new window


I don't have that many matches in my iptables

Chain fail2ban-block-bot-name (1 references)
num  target     prot opt source               destination
1    REJECT     all  --  100-43-91-119.spider.yandex.com  anywhere             reject-with icmp-port-unreachable
2    RETURN     all  --  anywhere             anywhere

Open in new window


Looking at the log file you can see another yandex bot came by too after I restarted fail2ban at 8:57 AM.

[root@ip-172-31-8-214 filter.d]# tail -100 /var/log/httpd/access_log | grep yandex
100.43.91.101 - - [29/Aug/2018:09:09:06 -0400] "GET /wp-content/uploads/2013/12/gooddog.gif HTTP/1.1" 200 4994 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
[root@ip-172-31-8-214 filter.d]# tail -500 /var/log/httpd/access_log | grep yandex
100.43.91.101 - - [29/Aug/2018:08:43:45 -0400] "GET /877-large_default/VS_C_TCM_Concentrate_Chinese_p_721.jpg HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
100.43.91.119 - - [29/Aug/2018:08:59:30 -0400] "GET /robots.txt HTTP/1.1" 200 99 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
100.43.91.101 - - [29/Aug/2018:09:09:06 -0400] "GET /wp-content/uploads/2013/12/gooddog.gif HTTP/1.1" 200 4994 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
[
[

Open in new window

0
nociSoftware EngineerCommented:
Looks good....
are the fail2ban-* chains still referenced from INPUT?

iptables --line-numbers -nv -L INPUT

Open in new window

0
sharingsunshineAuthor Commented:
[root@ip-172-31-8-214 fail2ban]# iptables --line-numbers -nv -L INPUT
Chain INPUT (policy ACCEPT 5598 packets, 559K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    10242 1109K fail2ban-block-bot-name  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
2    10235 1109K fail2ban-speed-move  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
3    10235 1109K fail2ban-login-multi  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
4    10235 1109K fail2ban-wordpress-soft  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
5    10133 1092K fail2ban-wordpress-hard  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
6    10133 1092K fail2ban-headless  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
7    12379 1307K fail2ban-BadBots  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443
8    16279  920K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set rukpuabucnro-block src
9     1776  103K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set ngtuczbrafal-block src

Open in new window

0
nociSoftware EngineerCommented:
So those are hit 10K+ times.
 I don't see the hitcounts in the earlier examples:

what does the following show?

iptables --line-numbers -nv -L fail2ban-block-bot-name
iptables --line-numbers -nv -L fail2ban-speed-move
iptables --line-numbers -nv -L fail2ban-BadBots

Open in new window


Looking again at your rules, the variouslog  lines need to have a short time between them... to count.
(1 second..., specify 1h if you want a match to count with another one in 1hour, that requires a recent fail2ban)
it is about findtime, make that either 3600 or a much longer time.

I am running 0.10.3 btw.
0
sharingsunshineAuthor Commented:
[root@ip-172-31-8-214 ~]# iptables --line-numbers -nv -L fail2ban-block-bot-name
Chain fail2ban-block-bot-name (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 REJECT     all  --  *      *       46.229.168.69        0.0.0.0/0            reject-with icmp-port-unreachable
2        0     0 REJECT     all  --  *      *       46.229.168.71        0.0.0.0/0            reject-with icmp-port-unreachable
3       20  1148 REJECT     all  --  *      *       100.43.91.124        0.0.0.0/0            reject-with icmp-port-unreachable
4       20  1148 REJECT     all  --  *      *       100.43.90.103        0.0.0.0/0            reject-with icmp-port-unreachable
5       20  1148 REJECT     all  --  *      *       100.43.81.100        0.0.0.0/0            reject-with icmp-port-unreachable
6       20  1148 REJECT     all  --  *      *       100.43.85.167        0.0.0.0/0            reject-with icmp-port-unreachable
7       25  1456 REJECT     all  --  *      *       100.43.81.104        0.0.0.0/0            reject-with icmp-port-unreachable
8       25  1448 REJECT     all  --  *      *       100.43.85.107        0.0.0.0/0            reject-with icmp-port-unreachable
9       22  1268 REJECT     all  --  *      *       100.43.90.117        0.0.0.0/0            reject-with icmp-port-unreachable
10      20  1152 REJECT     all  --  *      *       100.43.81.129        0.0.0.0/0            reject-with icmp-port-unreachable
11      20  1144 REJECT     all  --  *      *       100.43.85.108        0.0.0.0/0            reject-with icmp-port-unreachable
12       0     0 REJECT     all  --  *      *       46.229.168.74        0.0.0.0/0            reject-with icmp-port-unreachable
13       0     0 REJECT     all  --  *      *       46.229.168.79        0.0.0.0/0            reject-with icmp-port-unreachable
14       2   120 REJECT     all  --  *      *       46.229.168.77        0.0.0.0/0            reject-with icmp-port-unreachable
15      26  1512 REJECT     all  --  *      *       100.43.91.113        0.0.0.0/0            reject-with icmp-port-unreachable
16      23  1328 REJECT     all  --  *      *       100.43.91.105        0.0.0.0/0            reject-with icmp-port-unreachable
17       0     0 REJECT     all  --  *      *       46.229.168.73        0.0.0.0/0            reject-with icmp-port-unreachable
18      27  1576 REJECT     all  --  *      *       100.43.91.119        0.0.0.0/0            reject-with icmp-port-unreachable
19      30  1748 REJECT     all  --  *      *       100.43.91.101        0.0.0.0/0            reject-with icmp-port-unreachable
20       2   120 REJECT     all  --  *      *       46.229.168.78        0.0.0.0/0            reject-with icmp-port-unreachable
21       2   120 REJECT     all  --  *      *       46.229.168.65        0.0.0.0/0            reject-with icmp-port-unreachable
22       4   240 REJECT     all  --  *      *       46.229.168.81        0.0.0.0/0            reject-with icmp-port-unreachable
23       4   240 REJECT     all  --  *      *       46.229.168.76        0.0.0.0/0            reject-with icmp-port-unreachable
24       0     0 REJECT     all  --  *      *       46.229.168.84        0.0.0.0/0            reject-with icmp-port-unreachable
25      41  2460 REJECT     all  --  *      *       216.244.66.233       0.0.0.0/0            reject-with icmp-port-unreachable
26       2   120 REJECT     all  --  *      *       46.229.168.75        0.0.0.0/0            reject-with icmp-port-unreachable
27    289K   31M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
[root@ip-172-31-8-214 ~]#

Open in new window


Chain fail2ban-speed-move (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     289K   31M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
[root@ip-172-31-8-214 ~]#

Open in new window


[root@ip-172-31-8-214 ~]# iptables --line-numbers -nv -L fail2ban-BadBots
Chain fail2ban-BadBots (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 REJECT     all  --  *      *       185.24.233.19        0.0.0.0/0            reject-with icmp-port-unreachable
2     353K   37M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Open in new window

0
nociSoftware EngineerCommented:
So some rejects are hit multiple times....,  and sometimes a remote system only probes once or twice.
(And there are mutiple rejects in rules....).
If you meant  no new ones are added, do adjust the find time, your jail only seems to trigger when filters match multiple accesses within one second.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sharingsunshineAuthor Commented:
Your request to see my jails caused me to find a syntax error in the filter.  After making the change it began to block several ip's.  I still am having a problem with other jails that used to catch quite frequently all the marauders.  I can only conclude I need to upgrade 0.8.14 to 0.10.3.
1
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.