Fail2ban Stopped Jailing
Fail2ban stopped working. I have scoured the log files and no errors. It was working and it seems since my last Yum update for security it quit working. I can do fail2ban-client status and it shows 7 jails. I can look in the iptables and it shows the jails. However, when I run fail2ban-regex it shows many hits but none are getting blocked by the iptables. The iptables are on.
This is 0.8.14 and I am on a Linux 2 Amazon Ami with PHP 7 and Python 2.7.14.
When it was working it had over 221 ips banned just in one jail.
Please help me get this going. The bots overrun my system if it isn't in place.
This is 0.8.14 and I am on a Linux 2 Amazon Ami with PHP 7 and Python 2.7.14.
When it was working it had over 221 ips banned just in one jail.
Please help me get this going. The bots overrun my system if it isn't in place.
ASKER
it is running -
So how do I get it to jail the log hits?
root 27940 27.2 0.1 1235908 13516 ? Sl 22:24 1:39 /bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
root 29287 0.0 0.0 119464 924 pts/2 S+ 22:30 0:00 grep --color=auto fail2banSo how do I get it to jail the log hits?
Is the method of the jail changed? and still available?...
if the original fail2ban .conf files have been updated by you for use then they maybe overwritten by the install preventing some action.
For personal use and activatation of jails use *.local files not the *.conf files.
in recent fail2ban some changes have beenmade to activations, filters and jails so old jails might need some adjustments.
Besides that the fail2ban logging should give some indication about what it does.
if the original fail2ban .conf files have been updated by you for use then they maybe overwritten by the install preventing some action.
For personal use and activatation of jails use *.local files not the *.conf files.
in recent fail2ban some changes have beenmade to activations, filters and jails so old jails might need some adjustments.
Besides that the fail2ban logging should give some indication about what it does.
ASKER
nothing has changed related to Fail2Ban. I remembered my configuration was suspect on the 0.8 distribution. I think that is where the problem lies.
I haven't overwritten anything and it was working perfectly and then stopped. I found out because all of the bot activity crashed my instance.
I have looked into the fail2ban.log and didn't find anything. If you can tell me what to search for I will give it another go.
Thanks,
I haven't overwritten anything and it was working perfectly and then stopped. I found out because all of the bot activity crashed my instance.
I have looked into the fail2ban.log and didn't find anything. If you can tell me what to search for I will give it another go.
Thanks,
You're correct. Something has changed + now you have to track down what changed.
Follow my instructions above for debugging this.
Follow my instructions above for debugging this.
fail2ban-client -v -f start
should tell you which problems there are (you might need to stop fail2ban-server)
should tell you which problems there are (you might need to stop fail2ban-server)
ASKER
I have already stated that I have looked extensively in /var/log*. If you have a suggestion for a keyword to use that would be very helpful. I have searched for fail2ban, failed, error.
If you know any command line snippets they would be extremely helpful to know.
If you know any command line snippets they would be extremely helpful to know.
ASKER
Bernard, How do I know which are the problems?
Can you show a jail snippet and all filters & actions that are mentioned in it?
ASKER
[block-bot-name]
enabled = true
filter = block-bot-name
action = iptables[name=block-bot-name, port=443, protocol=tcp]
mail-buffered[name=block-bot-name, dest=ec2-user@localhost, sender=root@abc.com]
logpath = /path-to-log
maxretry = 1
bantime = 31556736
findtime =1# Fail2Ban configuration file
#
# Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# above mentioned bots.
[Definition]
failregex = ^<HOST> - - .*(pingdom\.com|ahrefs\.com|yandex\.com|dotbot\.com|semrush\.com)
ignoreregex =[root@ip-172-31-8-214 action.d]# cat iptables.conf
# Fail2Ban configuration file
#
# Author: Cyril Jaquier
#
#
[INCLUDES]
before = iptables-blocktype.conf
[Definition]
# Option: actionstart
# Notes.: command executed once at the start of Fail2Ban.
# Values: CMD
#
actionstart = iptables -N fail2ban-<name>
iptables -A fail2ban-<name> -j RETURN
iptables -I <chain> -p <protocol> --dport <port> -j fail2ban-<name>
# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
#
actionstop = iptables -D <chain> -p <protocol> --dport <port> -j fail2ban-<name>
iptables -F fail2ban-<name>
iptables -X fail2ban-<name>
# Option: actioncheck
# Notes.: command executed once before each actionban command
# Values: CMD
#
actioncheck = iptables -n -L <chain> | grep -q 'fail2ban-<name>[ \t]'
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionban = iptables -I fail2ban-<name> 1 -s <ip> -j <blocktype>
# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionunban = iptables -D fail2ban-<name> -s <ip> -j <blocktype>
[Init]
# Default name of the chain
#
name = default
# Option: port
# Notes.: specifies port to monitor
# Values: [ NUM | STRING ] Default:
#
port = ssh
# Option: protocol
# Notes.: internally used by config reader for interpolations.
# Values: [ tcp | udp | icmp | all ] Default: tcp
#
protocol = tcp
# Option: chain
# Notes specifies the iptables chain to which the fail2ban rules should be
# added
# Values: STRING Default: INPUT
chain = INPUT[root@ip-172-31-8-214 ~]# fail2ban-regex /var/log/httpd/access_log /etc/fail2ban/filter.d/block-bot-name.conf
Running tests
=============
Use failregex file : /etc/fail2ban/filter.d/block-bot-name.conf
Use log file : /var/log/httpd/access_log
Results
=======
Failregex: 231 total
|- #) [# of hits] regular expression
| 1) [231] ^<HOST> - - .*(pingdom\.com|ahrefs\.com|yandex\.com|dotbot\.com|semrush\.com)
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [62327] Day/MONTH/Year:Hour:Minute:Second
`-
Lines: 62327 lines, 0 ignored, 231 matched, 62096 missed
Missed line(s): too many to print. Use --print-all-missed to print all 62096 linesI don't have that many matches in my iptables
Chain fail2ban-block-bot-name (1 references)
num target prot opt source destination
1 REJECT all -- 100-43-91-119.spider.yandex.com anywhere reject-with icmp-port-unreachable
2 RETURN all -- anywhere anywhereLooking at the log file you can see another yandex bot came by too after I restarted fail2ban at 8:57 AM.
[root@ip-172-31-8-214 filter.d]# tail -100 /var/log/httpd/access_log | grep yandex
100.43.91.101 - - [29/Aug/2018:09:09:06 -0400] "GET /wp-content/uploads/2013/12/gooddog.gif HTTP/1.1" 200 4994 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
[root@ip-172-31-8-214 filter.d]# tail -500 /var/log/httpd/access_log | grep yandex
100.43.91.101 - - [29/Aug/2018:08:43:45 -0400] "GET /877-large_default/VS_C_TCM_Concentrate_Chinese_p_721.jpg HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
100.43.91.119 - - [29/Aug/2018:08:59:30 -0400] "GET /robots.txt HTTP/1.1" 200 99 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
100.43.91.101 - - [29/Aug/2018:09:09:06 -0400] "GET /wp-content/uploads/2013/12/gooddog.gif HTTP/1.1" 200 4994 "-" "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
[
[
Looks good....
are the fail2ban-* chains still referenced from INPUT?
are the fail2ban-* chains still referenced from INPUT?
iptables --line-numbers -nv -L INPUTASKER
[root@ip-172-31-8-214 fail2ban]# iptables --line-numbers -nv -L INPUT
Chain INPUT (policy ACCEPT 5598 packets, 559K bytes)
num pkts bytes target prot opt in out source destination
1 10242 1109K fail2ban-block-bot-name tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
2 10235 1109K fail2ban-speed-move tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
3 10235 1109K fail2ban-login-multi tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
4 10235 1109K fail2ban-wordpress-soft tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
5 10133 1092K fail2ban-wordpress-hard tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
6 10133 1092K fail2ban-headless tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
7 12379 1307K fail2ban-BadBots tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
8 16279 920K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 match-set rukpuabucnro-block src
9 1776 103K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 match-set ngtuczbrafal-block src
So those are hit 10K+ times.
I don't see the hitcounts in the earlier examples:
what does the following show?
Looking again at your rules, the variouslog lines need to have a short time between them... to count.
(1 second..., specify 1h if you want a match to count with another one in 1hour, that requires a recent fail2ban)
it is about findtime, make that either 3600 or a much longer time.
I am running 0.10.3 btw.
I don't see the hitcounts in the earlier examples:
what does the following show?
iptables --line-numbers -nv -L fail2ban-block-bot-name
iptables --line-numbers -nv -L fail2ban-speed-move
iptables --line-numbers -nv -L fail2ban-BadBotsLooking again at your rules, the variouslog lines need to have a short time between them... to count.
(1 second..., specify 1h if you want a match to count with another one in 1hour, that requires a recent fail2ban)
it is about findtime, make that either 3600 or a much longer time.
I am running 0.10.3 btw.
ASKER
[root@ip-172-31-8-214 ~]# iptables --line-numbers -nv -L fail2ban-block-bot-name
Chain fail2ban-block-bot-name (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 REJECT all -- * * 46.229.168.69 0.0.0.0/0 reject-with icmp-port-unreachable
2 0 0 REJECT all -- * * 46.229.168.71 0.0.0.0/0 reject-with icmp-port-unreachable
3 20 1148 REJECT all -- * * 100.43.91.124 0.0.0.0/0 reject-with icmp-port-unreachable
4 20 1148 REJECT all -- * * 100.43.90.103 0.0.0.0/0 reject-with icmp-port-unreachable
5 20 1148 REJECT all -- * * 100.43.81.100 0.0.0.0/0 reject-with icmp-port-unreachable
6 20 1148 REJECT all -- * * 100.43.85.167 0.0.0.0/0 reject-with icmp-port-unreachable
7 25 1456 REJECT all -- * * 100.43.81.104 0.0.0.0/0 reject-with icmp-port-unreachable
8 25 1448 REJECT all -- * * 100.43.85.107 0.0.0.0/0 reject-with icmp-port-unreachable
9 22 1268 REJECT all -- * * 100.43.90.117 0.0.0.0/0 reject-with icmp-port-unreachable
10 20 1152 REJECT all -- * * 100.43.81.129 0.0.0.0/0 reject-with icmp-port-unreachable
11 20 1144 REJECT all -- * * 100.43.85.108 0.0.0.0/0 reject-with icmp-port-unreachable
12 0 0 REJECT all -- * * 46.229.168.74 0.0.0.0/0 reject-with icmp-port-unreachable
13 0 0 REJECT all -- * * 46.229.168.79 0.0.0.0/0 reject-with icmp-port-unreachable
14 2 120 REJECT all -- * * 46.229.168.77 0.0.0.0/0 reject-with icmp-port-unreachable
15 26 1512 REJECT all -- * * 100.43.91.113 0.0.0.0/0 reject-with icmp-port-unreachable
16 23 1328 REJECT all -- * * 100.43.91.105 0.0.0.0/0 reject-with icmp-port-unreachable
17 0 0 REJECT all -- * * 46.229.168.73 0.0.0.0/0 reject-with icmp-port-unreachable
18 27 1576 REJECT all -- * * 100.43.91.119 0.0.0.0/0 reject-with icmp-port-unreachable
19 30 1748 REJECT all -- * * 100.43.91.101 0.0.0.0/0 reject-with icmp-port-unreachable
20 2 120 REJECT all -- * * 46.229.168.78 0.0.0.0/0 reject-with icmp-port-unreachable
21 2 120 REJECT all -- * * 46.229.168.65 0.0.0.0/0 reject-with icmp-port-unreachable
22 4 240 REJECT all -- * * 46.229.168.81 0.0.0.0/0 reject-with icmp-port-unreachable
23 4 240 REJECT all -- * * 46.229.168.76 0.0.0.0/0 reject-with icmp-port-unreachable
24 0 0 REJECT all -- * * 46.229.168.84 0.0.0.0/0 reject-with icmp-port-unreachable
25 41 2460 REJECT all -- * * 216.244.66.233 0.0.0.0/0 reject-with icmp-port-unreachable
26 2 120 REJECT all -- * * 46.229.168.75 0.0.0.0/0 reject-with icmp-port-unreachable
27 289K 31M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
[root@ip-172-31-8-214 ~]#Chain fail2ban-speed-move (1 references)
num pkts bytes target prot opt in out source destination
1 289K 31M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
[root@ip-172-31-8-214 ~]#[root@ip-172-31-8-214 ~]# iptables --line-numbers -nv -L fail2ban-BadBots
Chain fail2ban-BadBots (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 REJECT all -- * * 185.24.233.19 0.0.0.0/0 reject-with icmp-port-unreachable
2 353K 37M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Your request to see my jails caused me to find a syntax error in the filter. After making the change it began to block several ip's. I still am having a problem with other jails that used to catch quite frequently all the marauders. I can only conclude I need to upgrade 0.8.14 to 0.10.3.
Open in new window
If not, likely the following will work...
Open in new window
Then check /var/log/* entries for problems if fail2ban fails to restart.
I normally setup fail2ban to log to it's own log file...
Usually you can change this in fail2ban.conf logtarget=/var/log/fail2ba
At least this works on most Distros.