iptables hitcount, limit, log not working

Mark
Mark used Ask the Experts™
on
I have the iptables script shown below. The --hitcount and --limit rules are not working (lines 5, 10, 13). I just received 139 attempted ssh logins within 44 seconds. This is not uncommon. No message with "SSH Break-in attempt" is ever logged in either /var/log/messages or /var/log/syslog. Please advise on how to fix. I've posted this question on various web forums without getting a solution.

I am concerned with eth0 which is connected to the Internet. eth1 is local LAN traffic only having a few Windows workstations.

Linux slackware distro version 13.37.0, kernel 2.6.37.6, iptables version 1.40.10.

Script:
    iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
    iptables -A INPUT -i lo -j ACCEPT
    iptables -A INPUT -i eth1 -m state --state NEW -j ACCEPT

    iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 \
        --hitcount 10 --rsource -j LOG --log-prefix "SSH Break-in attempt" --log-level 6

    # The following blocks an IP for 60 seconds after 10 unsuccessful ssh attempts

    iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 \
        --hitcount 10 --rsource -j DROP

    iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -m state --state NEW -j ACCEPT

    iptables -A INPUT -p TCP -i eth0 -m multiport --dports 25,53,80,443 -m state --state NEW -j ACCEPT

    iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -m state --state NEW -j ACCEPT
    iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
    iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 --dport 37 -j ACCEPT

    iptables -A OUTPUT -o eth0 -m state --state NEW -j ACCEPT
    iptables -A OUTPUT -o lo -j ACCEPT
    iptables -A OUTPUT -o eth1 -m state --state NEW -j ACCEPT

Open in new window


iptables -L -v -n
Chain INPUT (policy DROP 6449 packets, 974K bytes)
 pkts bytes target     prot opt in     out     source               destination
    8   432 ACCEPT     tcp  --  *      *       64.129.23.99         0.0.0.0/0           tcp dpt:37
 117M  199G ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
 2586  155K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
20014 2265K ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           state NEW
    0     0 LOG        tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 state NEW recent: UPDATE seconds: 60 hit_count: 10 name: DEFAULT side: source LOG flags 0 level 6 prefix `SSH Break-in attempt'
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 state NEW recent: UPDATE seconds: 60 hit_count: 10 name: DEFAULT side: source
  305 17432 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02 limit: avg 1/sec burst 5 state NEW
 5428  289K ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           multiport dports 25,53,80,443 state NEW
   10   774 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           multiport dports 53 state NEW

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy DROP 38 packets, 2120 bytes)
 pkts bytes target     prot opt in     out     source               destination
  66M  138G ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
18009 1147K ACCEPT     all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           state NEW
 2586  155K ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
94575 6395K ACCEPT     all  --  *      eth1    0.0.0.0/0            0.0.0.0/0           state NEW

Open in new window

Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Duncan RoeSoftware Developer

Commented:
You are 2 Slackware revisions behind. But for iptables, that shouldn't matter.
You could make your filter more efficient by jumping to your own created LOGDROP chain in the line 5 rule rather than duplicating the tests in the line 10 rule.
If you are being spammed, it may be that  --rsource in the rule is letting the segments through - a real spammer is probably spoofing the source addresses. You could check this with tcpdump.
It may be that --state is not working for you. The man page says
The "state" extension is a subset of the "conntrack" module.  "state" allows access to the connection tracking state for this packet.
and for conntrack it says
This module, when combined with connection tracking, allows access to the connection tracking state for this packet/connection.
You only appear to have a filter table, so no connection tracking (which is part of nat AFAIK). I would be using TCP flags (e.g.SYN, FIN) instead.

Author

Commented:
Duncan Roe: > If you are being spammed, it may be that  --rsource in the rule is letting the segments through

So, should I remove the --rsource modifier? The man page says --rsource is the default, so how would I negate that?

As to possibly needing connection tracking for "state" to work, you are way over my head with your explanation. I am no iptables guru. I basically monkey-typed these rules from an assortment of google searches and EE questions over the past year.

I don't have *that* many rules. Could you possibly post an example of how you would implement working --hitcount, --limit, --state, connection tracking rules for what I am trying to do? I could probably reverse-engineer an idea of what's going on from an example.

Author

Commented:
btw - I've set up tcpdump to capture port 22 connection requests on eth0. We'll see if that gives any information:

tcpdump -n -i eth0 port 22 and 'tcp[13] & 2 = 2' and not host mySSHhost
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Duncan RoeSoftware Developer

Commented:
--rsource is the default, so removing it is a no-op. If you're really getting spoofed SYN segments, you may have to home in on some characteristic they have. Or maybe you can filter to only allow in the "known good" ones, assuming the attacker doesn't spoof their source addresses.
I've attached my filter table. It runs on the firewall / router, which is a separate system That's what you're supposed to do. The only traffic I allow into that system from the internet if dhcp & dns responses.
By defining my own chains, I eliminate the need for repetitive checking, as I alluded to earlier
rc.iptable-filter.txt
Duncan RoeSoftware Developer

Commented:
Regarding your tcpdump line, is mySSHhost your system?  If the calls are directed to it,  then not host mySSHhost will exclude them. Perhaps try not src host mySSHhost

Author

Commented:
I've not created a chain before. If I merge my lines 5/6 and 10/11 into the following, will that work to DROP IPs that attempt 10 connections within 1 minute? Is "-m state NEW" sufficient to trap on connection attempts or should I also include the --syn option?

iptables -N logdrop
iptables -A logdrop -p tcp --dport 22 -i eth0 -m state NEW -m recent --update -- seconds 60 \
        --hitcount 10 -j DROP

iptables -A logdrop -j LOG --log-level   6   --log-prefix "SSH Break-in attempt"

Not to be whinney, but everywhere I've posted this issue, the respondants focus entirely on the fact that I have two semi-duplicated lines that should be in a chain, and my main problem gets completely lost. So, let's go ahead and get the chain thing out of the way so I can figure out why my --limit and --hitcount options are not working. Your example script says "too much stuff gets logged" whereas I get absolutely nothing logged (that I can find).

If this chain setup suddenly makes things start working, then hallelujah! and I retract my grumble.

> Regarding your tcpdump line, is mySSHhost your system?  If the calls are directed to it,  then not host mySSHhost will exclude them. Perhaps try not src host mySSHhost

mySSHhost is a remote host, external to the system in question. I ssh into the target system. The tcpdump as I have it seems to work fine. I am directing tcpdump output to a file which I can peruse later. While tcpdump is running. I am tail -f'ing /var/log/messages and I can see the hackers attempting to log in while tcpdump shows the connection attempts. Interestingly, there seem to be many more "Failed password for invalid user" logs into /var/log/messages, than corresponding tcpdump output lines -- but that is a mystery for later.

> If you're really getting spoofed SYN segments, you may have to home in on some characteristic they have.

I don't think spoofed SYN segments are my initial concern. I have another cron job that scans the /var/log/messages file looking for these "Failed password for invalid user" message and runs an iptables ... -DROP on the subnet they are in. This job runs once per minute, and always blocks the intruder, so most of the IPs must not be spoofed at the level I am concerned with. Because of this cron job I rarely have log in attempts beyond one minute.

What I want is for iptables to block at 10 failed attempts, not rely on my cron job to block a minute later after 182 attempts.
Duncan RoeSoftware Developer

Commented:
The reason people focus on 2 almost identical lines is that they slow down your filter. It's simple enough when you "get it", which you haven't, yet. First, you need to define LOGDROP
iptables -N logdrop
iptables -A logdrop -j LOG --log-level   6   --log-prefix "SSH Break-in attempt"
iptables -A logdrop -j DROP

Open in new window

You change line 5/6 to be
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 \
  --hitcount 10 --rsource -j LOGDROP

Open in new window

You delete line 10/11. Do you see how it works now?
Now to why your rule is not matching on the traffic. I have never used -m recent, so can't for sure say how it works, but if the "source address" includes the source port then likely you'll never see 2 the same. There is only the SYN segment, so count stays at 1.
Here's another thing: how do you expect the addresses to be in the list at all? Only --set will put them there on my [re-]reading of the man page. --rcheck (and therefore --update) only tell you if it's in the list. They do not create list entries.
Duncan RoeSoftware Developer

Commented:
The more I look, the more I fell certain the last para in my previous post is your problem. What we need is to add an incoming candidate packet to the list if it is not there, and then check its hit count. (We don't really need to check the count for a just-added address, but doing anything else is too hard for me). You need to do this with 2 more new chains (doing it any other way just gets too messy). We need

1. A chain to test the hit count of each candidate packet

2. a chain to add a new address into the list

The only way you can conditionally perform an action is to jump to a chain containing that action, because the only conditional statement is JUMP. But the jumped-to chain can end with RETURN, which makes JUMP really a CALL. I don't want to do a --set for every segment because I suspect --set may reset the hit count. Define these chains after the LOGDROP chain but before any rules
iptables -N setaddr
iptables -A setaddr -m recent --set
iptables -A setaddr -J RETURN

iptables -N checkcount
iptables -A checkcount -m recent ! --update -J setaddr
iptables -A checkcount -m recent --hitcount 10 -J logdrop
iptables -A checkcount -J RETURN

Open in new window

Again, you need to change line 5/6
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -j checkcount

Open in new window

Author

Commented:
Based on your comments, I'll try the following:

# I assume we are still creating the logdrop chain since you reference it below.

iptables -N setaddr
iptables -A setaddr -m recent --set
iptables -A setaddr -J RETURN

iptables -N checkcount
iptables -A checkcount -m recent ! --update -J setaddr
iptables -A checkcount -m recent --hitcount 10 -J logdrop
iptables -A checkcount -J RETURN

iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -j checkcount

# and the remaining lines stay the same, right?

iptables -A INPUT -p TCP -i eth0 -m multiport --dports 25,53,80,443 -m state --state NEW -j ACCEPT

iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -m state --state NEW -j ACCEPT
iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 --dport 37 -j ACCEPT

iptables -A OUTPUT -o eth0 -m state --state NEW -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -o eth1 -m state --state NEW -j ACCEPT

Open in new window


I'll await your "nihil obstat" approval before I try running this ...

Author

Commented:
Sorry, meant to post more in previous question, but am having "fun" with the new EE layout.

I meant to past the logdrop rules at the beginning, but didn't, and I haven't figured out how to edit my posting yet (if that's still an option). So, pretend these lines are immediately following my " I assume we are still creating the logdrop chain" comment:

iptables -N logdrop
iptables -A logdrop -j LOG --log-level   6   --log-prefix "SSH Break-in attempt"
iptables -A logdrop -j DROP

In your posting 40043734 you had lines 5/6 changing to:

iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 \
  --hitcount 10 --rsource -j LOGDROP

but in your posting 40043781 you have lines 5/6 changing to:

iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -j checkcount

where is the --seconds 60 dealt with in your 2nd revision? In my consolidation in my previous posting I don't see where we are limiting to 10 attempts within 1 minutes. Also, where is the --limit of connection attempts to 1 per second? Are these somehow dealt with implicitly or are they not necessary?
Duncan RoeSoftware Developer

Commented:
My bad re 60s. I think it's all right just to put it back in the recent filter (the not (!) inverts update while 60s elaborates update).
I didn't change limit 1/s so didn't mention it - I was only seeking to post changes. The more recent version of line 5/6 is the one to use therefore. Here are your original rules with my suggested amendments
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth1 -m state --state NEW -j ACCEPT

iptables -N logdrop
iptables -A logdrop -j LOG --log-level 6 --log-prefix "SSH Break-in attempt"
iptables -A logdrop -j DROP

iptables -N setaddr
iptables -A setaddr -m recent --set
iptables -A setaddr -J RETURN

iptables -N checkcount
iptables -A checkcount -m recent ! --update --seconds 60 -J setaddr
iptables -A checkcount -m recent --hitcount 10 -J logdrop
iptables -A checkcount -J RETURN

iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -j checkcount
iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -m state --state NEW -j ACCEPT

iptables -A INPUT -p TCP -i eth0 -m multiport --dports 25,53,80,443 -m state --state NEW -j ACCEPT

iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -m state --state NEW -j ACCEPT
iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 --dport 37 -j ACCEPT

iptables -A OUTPUT -o eth0 -m state --state NEW -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -o eth1 -m state --state NEW -j ACCEPT

Open in new window

This is pretty much what you posted except I reinstated --seconds 60 and the limit rule. Also I inserted your original lines 1-3 and the logdrop chain. This is the diff
09:25:25$ diff -w newer_iptables_rules my_new_rules 
0a1,8
> iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
> iptables -A INPUT -i lo -j ACCEPT
> iptables -A INPUT -i eth1 -m state --state NEW -j ACCEPT
> 
> iptables -N logdrop
> iptables -A logdrop -j LOG --log-level 6 --log-prefix "SSH Break-in attempt"
> iptables -A logdrop -j DROP
> 
6c14
< iptables -A checkcount -m recent ! --update -J setaddr
---
> iptables -A checkcount -m recent ! --update --seconds 60 -J setaddr
10a19
> iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -m state --state NEW -j ACCEPT
09:25:40$ 

Open in new window

Duncan RoeSoftware Developer

Commented:
How do you get your chains to have policy DROP? Mine default to policy ACCEPT. Neither of us set a policy anywhere so I guess it must be that we have different distributions. I would get rid of your OUTPUT rules and set policy ACCEPT instead
iptables -P OUTPUT ACCEPT

Open in new window

man iptables says there's an implicit RETURN at the end of every user-defined chain, so having them there explicitly is just documentation. You could remove them if you want.

Author

Commented:
OK - I will try out your suggestion in posting 40055096 and get back with results ASAP.  I'll try the new rules at 0:00 tonight to be sure of what is effect. Thanks

Author

Commented:
First problem, when I run the -J rules, I get the following:
$ iptables -A setaddr -J RETURN
iptables v1.4.10: option `-J' requires an argument
Try `iptables -h' or 'iptables --help' for more information.

Open in new window

I don't find the -J option in `man iptables`. Is this an option related to your version of iptables, or did you mean -j (lowercase)? It seems there must be such an option if iptables says it "required an argument". However, you are specifying an argument: "RETURN", so I don't get what's going on.

Author

Commented:
> How do you get your chains to have policy DROP?

There are several additional rules at the beginning of my script that I left off for "clarity". Perhaps I shouldn't have, but I found that other postings generated debate on these with no consensus on whether they were good or bad. Sorry if they are important to you for what you are trying to construct. Note that I included them per recommendations from forum postings from a LONG time ago and I basically just monkey-typed them at the time with virtually no understanding. I just assumed they were standard initialization stuff. Here are the extra rules. You can see the default policies set in the first 3. If you think some of these are bogus, I can remove or alter.
iptables -P INPUT DROP
    iptables -P FORWARD DROP
    iptables -P OUTPUT DROP

    iptables -t nat -P PREROUTING ACCEPT
    iptables -t nat -P POSTROUTING ACCEPT
    iptables -t nat -P OUTPUT ACCEPT

    iptables -t mangle -P PREROUTING ACCEPT
    iptables -t mangle -P INPUT ACCEPT
    iptables -t mangle -P FORWARD ACCEPT
    iptables -t mangle -P OUTPUT ACCEPT
    iptables -t mangle -P POSTROUTING ACCEPT

    iptables -t raw -P PREROUTING ACCEPT
    iptables -t raw -P OUTPUT ACCEPT

    iptables -F
    iptables -F -t nat
    iptables -F -t mangle
    iptables -F -t raw

    iptables -X
    iptables -X -t nat
    iptables -X -t mangle
    iptables -X -t raw

Open in new window

Duncan RoeSoftware Developer

Commented:
My bad; -J should be -j (lower case). I had it in mind it should be upper - don't know why.
my-new-rules
Duncan RoeSoftware Developer

Commented:
I would get rid of your OUTPUT policy rule (on line 3). Then you could remove all your OUTPUT rules, which pretty much let everything through anyway. Every rule you get rid of speeds up your filter (except Policy rules).

Author

Commented:
better, but still an error:

$ iptables -A checkcount -m recent --hitcount 10 -j logdrop
iptables v1.4.10: recent: you must specify one of `--set', `--rcheck' `--update' or `--remove'
Try `iptables -h' or 'iptables --help' for more information.

> I would get rid of your OUTPUT policy rule
For this test, I'm just running the script you posted (now with lowercase -j). When I get this sorted out, I will remove the OUPUT rule, per your suggestion. I think the person I got that script from was trying to put iptables into a virgin state before running the real rules to make sure no OUPUT rules were lingering.
Duncan RoeSoftware Developer

Commented:
Yes he was putting iptables in a virgin state and you can leave that, but he also set some policies to DROP, which is not the default.
As to the error - you can add either --rcheck or --update. You just did a --update on the previous line so --rcheck should be fine. My bad there, I'd never considered the recent filter before, but the error message is quite explicit and looking at it now it seems kind of obvious. New line
iptables -A checkcount -m recent --rcheck --hitcount 10 -j logdrop

Open in new window

my-new-rules

Author

Commented:
OK, the new script runs and did not appear to mess anything up! I've got tcpdump running to look for offenders. I've listed the `iptables -L -v -n` below. I have a list of IP addresses I want to block. Previously, I blocked these using:

iptables -t filter -I INPUT 4 -s 121.10.172.0/16 -j DROP

I've changed that to inserting my additional DROPs after line 7 (below) which is ` iptables -A INPUT -i eth1 -m state --state NEW -j ACCEPT`. Doing '-I INPUT 5' does the trick. Is that the best place to put these DROPs or should I insert them as the very first rule?

$ iptables -L -v -n
Chain INPUT (policy ACCEPT 68 packets, 17533 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  *      *       64.129.23.99         0.0.0.0/0           tcp dpt:37
1751K 4516M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    6   945 ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           state NEW
    0     0 checkcount  tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 state NEW
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02 limit: avg 1/sec burst 5 state NEW
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           multiport dports 25,53,80,443 state NEW
    0     0 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           multiport dports 53 state NEW

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
1125K  338M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    4   240 ACCEPT     all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           state NEW
    0     0 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
   10   694 ACCEPT     all  --  *      eth1    0.0.0.0/0            0.0.0.0/0           state NEW

Chain checkcount (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 setaddr    all  --  *      *       0.0.0.0/0            0.0.0.0/0           !recent: UPDATE seconds: 60 name: DEFAULT side: source
    0     0 logdrop    all  --  *      *       0.0.0.0/0            0.0.0.0/0           recent: CHECK hit_count: 10 name: DEFAULT side: source
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain logdrop (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           LOG flags 0 level 6 prefix `SSH Break-in attempt'
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain setaddr (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0            all  --  *      *       0.0.0.0/0            0.0.0.0/0           recent: SET name: DEFAULT side: source
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Open in new window

Duncan RoeSoftware Developer

Commented:
First up: please use --line-numbers with iptables -L. Then you get rule numbers in the output and don't have to relate to (somewhat arbitrary) line numbers in your posts.

You must put DROP lines before the ACCEPT. Rules are obeyed in sequence until one of the terminating targets ACCEPT, DROP, QUEUE or RETURN is encountered (I made up terminating to describe these: no further rules in the chain obeying them are processed).

You seem to have done that (I didn't think you had at first, hence the para above. That's why I like --line-numbers ).

The stats show no connection attempts at all to port 22 - is that right? I really have my doubts about using --state as I mentioned before - I would be using -p tcp --syn instead of --state NEW here (and -p tcp ! --syn for the others).

Author

Commented:
> You must put DROP lines before the ACCEPT ... You seem to have done that

I think so. The DROPs go in after the ACCEPT for eth1 (line 7 - sorry, I'll use --line-numbers next time), but before the 1st accept for eth0 (line 8).

> The stats show no connection attempts at all to port 22 - is that right?

Entirely possible since I ran the `iptables -L -v -n` right after running the script.

>  I really have my doubts about using --state as I mentioned before - I would be using -p tcp --syn instead of --state NEW here (and -p tcp ! --syn for the others

Well, you might be right. see my results so far, at bottom. So, to change ... immediately below are the script lines with "--state NEW":
iptables -A INPUT -i eth1 -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -j checkcount
iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -m state --state NEW -j ACCEPT
iptables -A INPUT -p TCP -i eth0 -m multiport --dports 25,53,80,443 -m state --state NEW -j ACCEPT
iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -m state --state NEW -j ACCEPT
iptables -A OUTPUT -o eth0 -m state --state NEW -j ACCEPT
iptables -A OUTPUT -o eth1 -m state --state NEW -j ACCEPT

Open in new window

Would I change these to:
iptables -A INPUT -i eth1 -p tcp --syn -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -i eth0 -p tcp --syn -j checkcount
iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -p tcp --syn -j ACCEPT
iptables -A INPUT -p TCP -i eth0 -m multiport --dports 25,53,80,443 -p tcp --syn -j ACCEPT
iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -p tcp --syn -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --syn -j ACCEPT
iptables -A OUTPUT -o eth1 -p tcp --syn -j ACCEPT

Open in new window

I assume I'd also get rid of the "-m state" bit as well.

> (and -p tcp ! --syn for the others).

Where would that apply? On the RELATED,ESTABLISHED states? A la ...
iptables -A INPUT -p tcp !--syn -j ACCEPT
iptables -A OUTPUT -p tcp !--syn -j ACCEPT

Open in new window


Results so far. I have one attempt. Not sure the --limit or --hitcount are working (perhaps changing to -p tcp --syn will help). My tcpdump shows 20 attempts with 10 attempts within the 1-minute range at 9:29. Perhaps tcpdump shows connection attempts anyway and the 1 connection per second limit is not in effect at this level?
2014-05-15 09:21:23.153625 IP 108.227.160.138.39600 > 64.129.23.95.22: Flags [S], seq 1908642144, win 5840, options [mss 1460,sackOK,TS val 163088538 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:42.219719 IP 108.227.160.138.53400 > 64.129.23.95.22: Flags [S], seq 1200354569, win 5840, options [mss 1460,sackOK,TS val 163587560 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:43.787910 IP 108.227.160.138.53592 > 64.129.23.95.22: Flags [S], seq 1227025254, win 5840, options [mss 1460,sackOK,TS val 163589264 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:44.674366 IP 108.227.160.138.53672 > 64.129.23.95.22: Flags [S], seq 1245524309, win 5840, options [mss 1460,sackOK,TS val 163590149 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:45.570141 IP 108.227.160.138.53813 > 64.129.23.95.22: Flags [S], seq 1252796187, win 5840, options [mss 1460,sackOK,TS val 163591044 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:46.463007 IP 108.227.160.138.53926 > 64.129.23.95.22: Flags [S], seq 1277314162, win 5840, options [mss 1460,sackOK,TS val 163591938 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:47.352291 IP 108.227.160.138.54015 > 64.129.23.95.22: Flags [S], seq 1276266783, win 5840, options [mss 1460,sackOK,TS val 163592828 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:57.130573 IP 108.227.160.138.54276 > 64.129.23.95.22: Flags [S], seq 1288943025, win 5840, options [mss 1460,sackOK,TS val 163602581 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:57.172452 IP 108.227.160.138.54857 > 64.129.23.95.22: Flags [S], seq 1389562097, win 5840, options [mss 1460,sackOK,TS val 163602602 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:58.394196 IP 108.227.160.138.55184 > 64.129.23.95.22: Flags [S], seq 1464375003, win 5840, options [mss 1460,sackOK,TS val 163603869 ecr 0,nop,wscale 7], length 0
2014-05-15 09:29:59.284215 IP 108.227.160.138.55227 > 64.129.23.95.22: Flags [S], seq 1472162416, win 5840, options [mss 1460,sackOK,TS val 163604759 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:09.163856 IP 108.227.160.138.55452 > 64.129.23.95.22: Flags [S], seq 1490450165, win 5840, options [mss 1460,sackOK,TS val 163614622 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:09.231943 IP 108.227.160.138.56029 > 64.129.23.95.22: Flags [S], seq 1582136016, win 5840, options [mss 1460,sackOK,TS val 163614642 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:15.168284 IP 108.227.160.138.56029 > 64.129.23.95.22: Flags [S], seq 1582136016, win 5840, options [mss 1460,sackOK,TS val 163620642 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:21.216906 IP 108.227.160.138.56547 > 64.129.23.95.22: Flags [S], seq 1678302213, win 5840, options [mss 1460,sackOK,TS val 163626663 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:27.284519 IP 108.227.160.138.57145 > 64.129.23.95.22: Flags [S], seq 1768466604, win 5840, options [mss 1460,sackOK,TS val 163632682 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:33.275290 IP 108.227.160.138.57684 > 64.129.23.95.22: Flags [S], seq 1853600027, win 5840, options [mss 1460,sackOK,TS val 163638703 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:39.310926 IP 108.227.160.138.58221 > 64.129.23.95.22: Flags [S], seq 1962461460, win 5840, options [mss 1460,sackOK,TS val 163644723 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:39.449472 IP 108.227.160.138.58679 > 64.129.23.95.22: Flags [S], seq 2056019320, win 5840, options [mss 1460,sackOK,TS val 163644742 ecr 0,nop,wscale 7], length 0
2014-05-15 09:30:45.271741 IP 108.227.160.138.58679 > 64.129.23.95.22: Flags [S], seq 2056019320, win 5840, options [mss 1460,sackOK,TS val 163650742 ecr 0,nop,wscale 7], length 0

Open in new window

/var/log/messages shows attempted logins from this IP at no more than once per second, so perhaps tcpdump logs things ahead of the firewall block, yes?
May 15 09:29:43 webserver sshd[27483]: Failed password for invalid user root from 108.227.160.138 port 53400 ssh2
May 15 09:29:44 webserver sshd[27485]: Failed password for invalid user root from 108.227.160.138 port 53592 ssh2
May 15 09:29:45 webserver sshd[27487]: Failed password for invalid user root from 108.227.160.138 port 53672 ssh2
May 15 09:29:46 webserver sshd[27489]: Failed password for invalid user root from 108.227.160.138 port 53813 ssh2
May 15 09:29:47 webserver sshd[27491]: Failed password for invalid user root from 108.227.160.138 port 53926 ssh2
May 15 09:29:48 webserver sshd[27493]: Failed password for invalid user root from 108.227.160.138 port 54015 ssh2
May 15 09:29:58 webserver sshd[27496]: Failed password for invalid user andi from 108.227.160.138 port 54857 ssh2
May 15 09:29:59 webserver sshd[27499]: Failed password for invalid user markus from 108.227.160.138 port 55184 ssh2
May 15 09:30:00 webserver sshd[27501]: Failed password for invalid user damon from 108.227.160.138 port 55227 ssh2

Open in new window

Unfortunately for this test, the ip blocking script ran on the minute and blocked this IP after 9:30, so I don't know if there would have been a lot more attempts. This IP attacked another computer in the system which does not have this script running and had no more attempts than this test host. Therefore, not sure if --hitcount is working. I think I will disable the blocking script for the time being and see what the iptables rules do.

Let me know if I've got the correct syntax on those -p tcp --syn rules and I'll put those in right away.

Lastly, I do have numerous "SSH Break-in" log messages (progress there!), but all for a legitimate external backup host that is trying to run rsync to copy files. Not sure why rsync wants to do so many SYN requests, but whatever. This IP is not blocked, probably because it never does more than 5 attempts per minute. I suppose this situation is slowing down the rsync. How would I permit this host through without logging messages?

iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 --dport 22 -j ACCEPT   (?)

I already have a rule for this host permitting port 37, so could I combine them as:

iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 --dports 22,37 -j ACCEPT
Duncan RoeSoftware Developer

Commented:
Perhaps tcpdump shows connection attempts anyway YES! it does!
I noticed that a while ago. tcpdump gets to see traffic earlier on than iptables, so you do see incoming stuff that is subsequently dropped

Author

Commented:
OK, fine for tcpdump. Thanks. Actually, the tcpdump log show recent attempts from several places that were blocked 2 weeks ago!

What about my question on permitting host 64.129.23.99 through without logging the "SSH Break-in attempt" message?  (last few sentences of my last posting)

Also, what about my question on the correct -p tcp --syn syntax? (2nd and 3rd code blocks of my last posting)

Author

Commented:
The most recent assailant shows 13 attempts within 4 seconds, so the --limit isn't working. I'm hoping the -p tcp --syn will address that.
May 15 21:46:15 webserver sshd[25762]: Failed password for invalid user admin from 191.236.25.16 port 1168 ssh2
May 15 21:46:16 webserver sshd[25764]: Failed password for invalid user root from 191.236.25.16 port 1169 ssh2
May 15 21:46:16 webserver sshd[25766]: Failed password for invalid user guest from 191.236.25.16 port 1170 ssh2
May 15 21:46:16 webserver sshd[25768]: Failed password for invalid user ubnt from 191.236.25.16 port 1171 ssh2
May 15 21:46:17 webserver sshd[25770]: Failed password for invalid user support from 191.236.25.16 port 1172 ssh2
May 15 21:46:17 webserver sshd[25772]: Failed password for invalid user test from 191.236.25.16 port 1173 ssh2
May 15 21:46:17 webserver sshd[25774]: Failed password for invalid user admin from 191.236.25.16 port 1174 ssh2
May 15 21:46:17 webserver sshd[25776]: Failed password for invalid user ftpuser from 191.236.25.16 port 1176 ssh2
May 15 21:46:18 webserver sshd[25778]: Failed password for invalid user vyatta from 191.236.25.16 port 1177 ssh2
May 15 21:46:18 webserver sshd[25780]: Failed password for invalid user pi from 191.236.25.16 port 1178 ssh2
May 15 21:46:18 webserver sshd[25782]: Failed password for invalid user user from 191.236.25.16 port 1179 ssh2
May 15 21:46:18 webserver sshd[25784]: Failed password for invalid user www from 191.236.25.16 port 1180 ssh2
May 15 21:46:19 webserver sshd[25786]: Failed password for invalid user PlcmSpIp from 191.236.25.16 port 1181 ssh2

Open in new window

Duncan RoeSoftware Developer

Commented:
I had an appointment today and dashed off the tcpdump response just before I left. It's been a long day.

For --syn, you must have a space between exclamation mark and minus
so your example should look like
iptables -A INPUT -p tcp ! --syn -j ACCEPT
iptables -A OUTPUT -p tcp ! --syn -j ACCEPT

Open in new window

Remove the OUTPUT rules
Chain OUTPUT has policy ACCEPT now.
Update to 64.129.23.99 looks OK
regarding http:#a40069256
Was anything logged?
How many SYN segments did tcpdump show?
Next time, please run your iptables -L command and check whether the rules you suspect did in fact get packet matches.

Author

Commented:
> •Remove the OUTPUT rules ... Chain OUTPUT has policy ACCEPT now

"policy ACCEPT" because I'm not running those iptables initialization rules. first things first! But when I reinstate those rules, I'll remove the output ones.

> •Update to 64.129.23.99 looks OK

Actually, it wasn't. I needed to put -m multiport ahead of the --dports. BTW what is "-m multiport"? I couldn't find a reference to it in the man page.

Also, I already had -p tcp on many of the rules, so adding -p tcp --syn gave an error. I removed the existing -p tcp. I also found that the -p tcp --syn options had to come before the --dport option or I got an error.

I commented out the rule for port 53. That has -p upd, so I was not sure if -p tcp --syn would work. What do you think:

iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -p tcp --syn -j ACCEPT

> Was anything logged?

Yes, but only for that host 64.129.23.99 which is one of our and was doing a long rsync. 248 messages logged in /var/log/messages, e.g.:
May 16 11:11:36 webserver kernel: [890842.237690] SSH Break-in attemptIN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:94:de:80:b9:c7:4c:08:00 SRC=64.129.23.99 DST=64.129.23.95 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48529 DF PROTO=TCP SPT=33766 DPT=22 WINDOW=4380 RES=0x00 SYN URGP=0

Open in new window


There was nothing in my tcpdump log because I excluded this host.

Question - since iptables logged these SYN connections, why didn't it DROP the connection to this host? The host ran the rsync job to completion.

> Next time, please run your iptables -L command and check whether the rules you suspect did in fact get packet matches.

I'll let it run for a while to accumulate some counts, then I'll post.

Author

Commented:
here's my current iptables -V, secondary DROPs removed for clarity:
$ iptables -V -l -n --line-numbers
Chain INPUT (policy ACCEPT 9109 packets, 1576K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    2412K  131M ACCEPT     tcp  --  *      *       64.129.23.99         0.0.0.0/0           tcp multiport dports 22,37
2    4991K 2170M ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:!0x17/0x02
3      227 13620 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
4      387 21384 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x02

[deleted]

102      3   136 checkcount  tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02
103      3   136 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02 limit: avg 1/sec burst 5
104   2096  109K ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x02 multiport dports 25,53,80,443

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 4636K packets, 15G bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain checkcount (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        3   136 setaddr    all  --  *      *       0.0.0.0/0            0.0.0.0/0           !recent: UPDATE seconds: 60 name: DEFAULT side: source
2        0     0 logdrop    all  --  *      *       0.0.0.0/0            0.0.0.0/0           recent: CHECK hit_count: 10 name: DEFAULT side: source
3        3   136 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain logdrop (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           LOG flags 0 level 6 prefix `SSH Break-in attempt'
2        0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain setaddr (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        3   136            all  --  *      *       0.0.0.0/0            0.0.0.0/0           recent: SET name: DEFAULT side: source
2        3   136 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Open in new window

and current script:
iptables -A INPUT -p tcp ! --syn -j ACCEPT      # RELATED,ESTABLISHED
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --syn -j ACCEPT

iptables -N logdrop
iptables -A logdrop -j LOG --log-level 6 --log-prefix "SSH Break-in attempt"
iptables -A logdrop -j DROP

iptables -N setaddr
iptables -A setaddr -m recent --set
iptables -A setaddr -j RETURN

iptables -N checkcount
iptables -A checkcount -m recent ! --update --seconds 60 -j setaddr
iptables -A checkcount -m recent --rcheck --hitcount 10 -j logdrop
iptables -A checkcount -j RETURN

iptables -A INPUT -p tcp --syn --dport 22 -i eth0 -j checkcount
iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -j ACCEPT

iptables -A INPUT -i eth0 -p tcp --syn -m multiport --dports 25,53,80,443 -j ACCEPT

# iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -p tcp --syn -j ACCEPT
iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 -m multiport --dports 22,37 -j ACCEPT

Open in new window

Duncan RoeSoftware Developer

Commented:
BTW what is "-m multiport"? I couldn't find a reference to it in the man page.
NO WONDER you are having a hard time with your filter. YOU ARE READING THE WRONG MAN PAGE
man iptables-extensions describes all the facilities you are trying to use. As system administrator (of CourtScan by any chance? ;) you owe it to yourself to understand this page completely. For instance:-
OF COURSE --syn and --dport have to be grouped together after -p tcp (-m tcp should be in there as well btw)
OF COURSE you needed to put -m multiport ahead of the --dports

Read man iptables-extensions. Read it again, and again until you understand it fully. If there's any item you still don't understand after making an honest attempt, I will do my best to answer any question you may have about it.

Once you tell me you fully understand what you are doing, I will take other questions again. If it looks to me that you should have known an answer from the man page, I will answer by advising you to Read The Fine Manual.
Duncan RoeSoftware Developer

Commented:
As a systems administrator, it also behooves you to have at least some rudimentary understanding of IP networking. Mixing -p tcp and -p udp, as you proposed for port 53 (DNS), is complete nonsense. You can read about the difference between TCP & UDP here or Google difference between TCP and UDP for many more
Duncan RoeSoftware Developer

Commented:
When I first saw your Q, I must say I thought twice before posting. My thinking was "here's another guy who wants to build a sophisticated iptables filter but has really no idea what he's taking on: I'm not going to get stuck answering every last little thing again".
But I did post, and you are making progress. Now you have the right documentation, you might well get there with no extra help from me.

Author

Commented:
man iptables-extensions  -  I don't have this manpage on either my slackware systems or my up-to-date Debian system. I tried apt-get install iptables-extensions on debian, "unable to locate". Where do you find this manpage?

> Mixing -p tcp and -p udp, as you proposed for port 53 (DNS), is complete nonsense.

Yes, I undersand the difference between tcp and upd, which is why I commented out the line. That was the result of a global file substitution of /-m state -- state NEW/ for /-p tcp --syn/. I posted that more as a question of using --syn with UDP. Perhaps it would have been more palatable had I asked if `iptables -A INPUT -p UDP --syn -i eth0 -m multiport --dports 53 -j ACCEPT` would work.

It doesn't strike me as that idiotic not knowing that "--syn and --dport have to be grouped together after -p tcp"

In my posting 40067689 I did list the port 53 rule and the other 7 rules you commented "OF COURSE" to before implement them and asked you to "let me know if I've got the correct syntax on those -p tcp --syn rules." You only commented that I needed a space after the ! on the ! --syn rules. I figured out the correct order myself when the rules didn't work.

I'm sorry you want to withdraw from this question. No, I am not well versed in iptables and yes, I do need guidance on this. I've been a Linux and WIndows SBS system administrator for many years and administer email (configuring sendmail and Exchange, spam filters, etc.), networks, websites, etc. I get a lot of help from EE when I need it. I figure a lot of stuff out on my own from manpages, googling, etc. I post here only when I fail to find what I need and need additional help. I've spent several months searching out answers to my problem on this issue and nothing I've tried works -- including this attempt so far.

I didn't think this would be a complicated request. My current script is only 17 lines long, including your additions on setting up the chains. Before starting this question my then-script did permit only the desired ports. My request in this message was about limiting connection attempts to a max of 1/second and 10/per minute. Despite all the time we've spent on this question, that still hasn't been accomplished and I'm doing nothing more than what I started with.

As you say, perhaps I'm almost there, so if you're interested in continuing, here are my latest results. Remember, my primary objectives are limit port 22 connections to once per second and limiting failed attempts to 10 per minute.

The --limit does not appear be working because I see many connection attempts within 1 minute, example:
May 17 11:38:50 webserver sshd[11301]: Failed password for invalid user root from 115.238.236.92 port 28857 ssh2
May 17 11:38:50 webserver sshd[11300]: Failed password for invalid user root from 115.238.236.92 port 28859 ssh2
May 17 11:38:50 webserver sshd[11298]: Failed password for invalid user root from 115.238.236.92 port 28853 ssh2
May 17 11:38:50 webserver sshd[11299]: Failed password for invalid user root from 115.238.236.92 port 28862 ssh2
May 17 11:38:50 webserver sshd[11301]: Failed password for invalid user root from 115.238.236.92 port 28857 ssh2
May 17 11:38:50 webserver sshd[11298]: Failed password for invalid user root from 115.238.236.92 port 28853 ssh2
May 17 11:38:50 webserver sshd[11300]: Failed password for invalid user root from 115.238.236.92 port 28859 ssh2
May 17 11:38:50 webserver sshd[11299]: Failed password for invalid user root from 115.238.236.92 port 28862 ssh2
May 17 11:38:50 webserver sshd[11298]: Failed password for invalid user root from 115.238.236.92 port 28853 ssh2
May 17 11:38:50 webserver sshd[11301]: Failed password for invalid user root from 115.238.236.92 port 28857 ssh2
May 17 11:38:50 webserver sshd[11300]: Failed password for invalid user root from 115.238.236.92 port 28859 ssh2
May 17 11:38:50 webserver sshd[11299]: Failed password for invalid user root from 115.238.236.92 port 28862 ssh2
May 17 11:38:50 webserver sshd[11301]: Failed password for invalid user root from 115.238.236.92 port 28857 ssh2
May 17 11:38:50 webserver sshd[11300]: Failed password for invalid user root from 115.238.236.92 port 28859 ssh2

Open in new window

Likewise, the --hitcount doesn't appear to be working to limit/DROP after 10 connection attempts in 1 minute. The example shown above had 34 attempts within one minute. Another recent attempt had 124 connection attempts within 5 minutes.

Do you have any idea why these rules are not working?

Author

Commented:
found iptables-extensions at http://ipset.netfilter.org/iptables-extensions.man.html. Not sure why neither of my Linux distros ships with this. I will review.
Duncan RoeSoftware Developer

Commented:
Hi Mark I don't want to withdraw. I do want to help you get yourself up to speed.

Where do you find this manpage?
In Slackware 14.1,  /usr/man/man8/iptables-extensions.8.gz is in package iptables-1.4.20-i486-1 or iptables-1.4.20-x86_64-1. Slackware 14.0 had iptables-1.4.14 which still had everything in the iptables man page. However I can find multiport in that page. Rather than searching for  -m multiport, you should have searched for plain multiport. The man page documents -m (or --match) precisely once, right at the top under MATCH EXTENSIONS. After that, there is a list of match extensions of which multiport is just one.

From what you have said, it looks like you are running a version of iptables prior to 1.4.20. Otherwise you would have the new man page. You might like to upgrade but I don't think that's your root problem.
Duncan RoeSoftware Developer

Commented:
I undersand the difference between tcp and upd, ...
Maybe, but did you understand that UDP is stateless? I didn't pick up that you tried to use -m state with UDP, but it became obvious when you tried to use --syn because the flags byte (implementing SYN, ACK, FIN &c.) is a part of Transmission Control so does not exist in the UDP header. I class that as "rudimentary knowledge" but others might differ...
Duncan RoeSoftware Developer

Commented:
It doesn't strike me as that idiotic not knowing that "--syn and --dport have to be grouped together after -p tcp"

I didn't say it was idiotic. But ...  --syn and --dport are documented together as part of the tcp match extension. You can put them in either order - it makes no difference. But you have to put them together. If you are on top of the man page you know that stuff.

The tcp match extension is unusual in that you don't need to preface the tests with -m tcp as long us you put them straight after -p tcp. I imagine this is for historical compatibility.
Duncan RoeSoftware Developer

Commented:
In my posting 40067689 ... I figured out the correct order myself when the rules didn't work.
Rather than thinking of it as the order you must switch to thinking of getting the grouping right. There are match extensions, and tests within each extension. Each match extension decides for itself what order to do its tests. If a rule has multiple extensions, each extension is executed in the order you have them so ordering can be important, especially for efficiency.

Rather than ask if syntax is correct, please just try it for yourself. Surely you have the odd test system?
Duncan RoeSoftware Developer

Commented:
Do you have any idea why these rules are not working?

I see 2 possibilities: 1. a previous rule accepted the packet; 2. the rule does not match the packet.
To resolve which of the above 2 is the correct hypothesis, I would append an INPUT rule which logs every packet that is being accepted by default. This will likely give you heaps of output to start with: whittle this down by inserting rules to ACCEPT the uninteresting packets (-p icmp comes to mind). If I were doing this, I'd be tailing the log file in an xterm somewhere and inserting the rules interactively (from a script session so I can afterwards look at what I did). That's what I call "being up to speed". YMMV

Author

Commented:
My iptables vesion is 1.4.10. I can look into upgrading. I know that -m mean match, I did not search for "-m multiport", just "multiport". I finally did find it in my manpage. I guess I must have fooped the spelling repeatedly.

>  SYN, ACK, FIN &c. is a part of Transmission Control so does not exist in the UDP header. I class that as "rudimentary knowledge"

Once upon a time I wrote programs that connected at the tcp/udp level, but that was a long time ago and I've really forgotten the details, so no I didn't recall that SYN was tcp not udp (but I'm getting refreshed!). My dealing with upd/tcp nowadays with respect to iptables has to do with knowing what services uses what protocol so I can set the right protocol in the rule. My research showed that DNS servers bind to UDP port 53. That's as far as I took it. I've not had to deal with packet-level message headers up to now. Anyway, this host is not acting as a DNS server, so I'm removing that rule.

> --syn and --dport are documented together as part of the tcp match extension

This may damage my credibility further, but I'm just now noticing that the double-dash options are sub-options to the single-dash options; not a normal Linux command construct where args are not typically position dependent (though some are). I guess never really noticed the pattern before until something broke. But, I did get those straightened out on my own!

> Rather than ask if syntax is correct, please just try it for yourself. Surely you have the odd test system?

Yes, but I am doing all this remotely and I didn't want to accidently cut off my own access (which I've done before) and have to physical get in the car and drive to the location to reset. Asking about syntax before I try something major can save me some pain. As a precaution, I run a cron job 20 minutes hence that will clear all rules. If my change does not block me, I kill the job. Otherwise, I wait 20 minutes. Nevertheless, I will refrain from this double-checking practice -- I've not managed to block myself during this project.

However, having said that, I am not that confident about constructing a logging rule, so, in response to your comment, "I would append an INPUT rule which logs every packet that is being accepted by default." I've added the rule:

iptables -A INPUT -i eth0 -p tcp -j LOG --log-level 6 --log-prefix "iptables Accepted "

I am tailing the logfile. I would have expected to see logging of all my remote ssh activity, but after several minutes, there is nothing in the logfile.

Also, I just had an attempted login of 28 attempts within 1 minute. No "iptables Accepted" messages. So my log rule must not be right. Wait ... after another 10 minutes I have 1 message:
May 18 07:17:59 webserver kernel: [1049425.746515] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=112.216.163.130 DST=64.129.23.95 LEN=40 TOS=0x00 PREC=0x00 TTL=106 ID=256 PROTO=TCP SPT=6000 DPT=1723 WINDOW=16384 RES=0x00 SYN URGP=0

Open in new window

Interestingly, this IP is not recorded in my tcpdump log (I'm recording SYN only) and is not otherwise recorded in /var/log/messages, so no actual ssh login attempt. I don't get it. Why this message and none of the 28 login attempts and none of my own ssh session, even on a new ssh session? Like I said, my log rule must be wrong.

If that's not enough for one posting ...

Note that I did get 4 of the "SSH Break-in attempt" message 2 days ago. This was for the same IP over a 7 second range. This was for an attempted login of 124 attempts over 5 minute period. The "SSH Break-in attempt" messages were logged *after* the 5 minutes of the 124 login attempt messages! This raises more questions:

Why does the logging occur after 124 attempts, not after 10?

Why is the logging part sort-of working but the DROP part is not? It seems to me that after logging the 1st message, that IP should be blocked; I shouldn't have gotten 3 more messages.

Why no more log messages for 2 days? I've certainly had plenty of attempts Does some counter need to be reset? I'm going restart the firewall and see if new logging occurs.

Author

Commented:
Now I've got 4 "accepted" messages:
May 18 07:17:59 webserver kernel: [1049425.746515] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=112.216.163.130 DST=64.129.23.95 LEN=40 TOS=0x00 PREC=0x00 TTL=106 ID=256 PROTO=TCP SPT=6000 DPT=1723 WINDOW=16384 RES=0x00 SYN URGP=0
May 18 07:37:05 webserver kernel: [1050570.600134] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=122.224.7.203 DST=64.129.23.95 LEN=40 TOS=0x08 PREC=0x20 TTL=107 ID=256 PROTO=TCP SPT=36660 DPT=2222 WINDOW=16384 RES=0x00 SYN URGP=0
May 18 07:49:30 webserver kernel: [1051314.298539] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=114.215.113.81 DST=64.129.23.95 LEN=40 TOS=0x00 PREC=0x00 TTL=104 ID=256 PROTO=TCP SPT=6000 DPT=2967 WINDOW=16384 RES=0x00 SYN URGP=0
May 18 08:01:17 webserver kernel: [1052021.143928] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=82.221.107.33 DST=64.129.23.95 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=48834 PROTO=TCP SPT=56016 DPT=5900 WINDOW=1024 RES=0x00 SYN URGP=0

Open in new window

but, if I append this rule

iptables -A INPUT -i eth0 -p tcp -j LOG --log-level 6 --log-prefix "iptables Accepted "

to the end of my INPUT chain, doesn't that mean it will pick up any "leftover" requests NOT handled by a preceeding rule? So, if I want to see what's sneaking by my LOG rules, wouldn't I have to put it after

iptables -A INPUT -p tcp --syn --dport 22 -i eth0 -j checkcount

and before

iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -j ACCEPT

Next ...

regarding --limit and --hitcount apparently not working, you wrote:

> I see 2 possibilities: 1. a previous rule accepted the packet; 2. the rule does not match the packet.

Since I did not run my iptables initialization commands with this test script, my INPUT default is ACCEPT. That might be the rule you mention as option 1. I have now changed the script to set the INPUT and FORWARD chains to default to DROP, and I've not touched the OUTPUT chain, per our previous discussion. Let's see if that does the trick.
Duncan RoeSoftware Developer

Commented:
(very late here) port 6000 is X (as in X Windows). Did you want an answer to any of the above? Going to bed now.
Your LOG rule looks all right to my sleepy eyes but are you only interested in TCP? I had a UDP DOS attack recently (to the NTP port, which is accepted and routed).

Also, are you running these firewall rules on the actual server system? It's recommended to have a router / firewall dedicated system with the servers on a protected LAN behind it.

I would stick with the INPUT default ACCEPT and drop packets after logging them if you want. You know what's getting dropped that way

Author

Commented:
I had to take down all the rules. I found that with these rules in place I was not receiving any mail, even though INPUT rule 7 has port 25. Do you see anything obvious? Maybe I should reinstate the RELATED,ESTABLISHED rule?
Chain INPUT (policy DROP 3 packets, 984 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  *      *       64.129.23.99         0.0.0.0/0           tcp multiport dports 22,37
2    10567  586K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:!0x17/0x02
3        0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
4        0     0 ACCEPT     tcp  --  eth1   *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x02
5        0     0 checkcount  tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02
6        0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02 limit: avg 1/sec burst 5
7        2    88 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x02 multiport dports 25,53,80,443

Chain FORWARD (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 5910 packets, 27M bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain checkcount (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 setaddr    all  --  *      *       0.0.0.0/0            0.0.0.0/0           !recent: UPDATE seconds: 60 name: DEFAULT side: source
2        0     0 logdrop    all  --  *      *       0.0.0.0/0            0.0.0.0/0           recent: CHECK hit_count: 10 name: DEFAULT side: source
3        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain logdrop (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           LOG flags 0 level 6 prefix `SSH Break-in attempt'
2        0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain setaddr (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0            all  --  *      *       0.0.0.0/0            0.0.0.0/0           recent: SET name: DEFAULT side: source
2        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Open in new window

Author

Commented:
Yeah, that seems to have helped. So I removed the revised:

iptables -A INPUT -p tcp ! --syn -j ACCEPT

and put back the old

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

email is now working again.
Duncan RoeSoftware Developer

Commented:
I can't see anything obvious.
To investigate further, I would reinstate the LOG message at the end of the INPUT chain. I assume you already checked your logs and there was no mention of dropping SMTP (port 25).
If you make INPUT policy ACCEPT, mail should keep working while you investigate.

I think it would be illuminating for both of us to get an answer to this - please pursue it.

N.B. policy DROP will drop UDP and ICMP, which could be the root cause. You really should do something about these protocols, if you are going to persist with policy DROP (which I do not recommend).

Also your LOG line should be
-A INPUT -i eth0 -j LOG --log-level 6 --log-prefix "iptables drop-through "

Open in new window

i.e. miss out -p tcp. We need to see everything that's being thrown away

Author

Commented:
Results so far:

As I mentioned, I changed the INPUT default to DROP, which I really, really hoped would be that "previous rule accepted the packet" condition you mentioned. Unfortunately, the --limit and --hitcount still don't appear to be working. I started my revised script at 10:37, but at 12:26 another attacker manged to get 31 attempts in within 4 seconds; so violating --limit AND --hitcount. Here are the /var/log/message for just the Failed Password logs:
May 18 12:26:00 webserver sshd[8545]: Failed password for invalid user root from 193.107.17.72 port 29249 ssh2
May 18 12:26:00 webserver sshd[8546]: Failed password for invalid user root from 193.107.17.72 port 29316 ssh2
May 18 12:26:00 webserver sshd[8549]: Failed password for invalid user root from 193.107.17.72 port 29455 ssh2
May 18 12:26:01 webserver sshd[8545]: Failed password for invalid user root from 193.107.17.72 port 29249 ssh2
May 18 12:26:01 webserver sshd[8546]: Failed password for invalid user root from 193.107.17.72 port 29316 ssh2
May 18 12:26:01 webserver sshd[8549]: Failed password for invalid user root from 193.107.17.72 port 29455 ssh2
May 18 12:26:01 webserver sshd[8545]: Failed password for invalid user root from 193.107.17.72 port 29249 ssh2
May 18 12:26:01 webserver sshd[8546]: Failed password for invalid user root from 193.107.17.72 port 29316 ssh2
May 18 12:26:01 webserver sshd[8549]: Failed password for invalid user root from 193.107.17.72 port 29455 ssh2
May 18 12:26:01 webserver sshd[8545]: Failed password for invalid user root from 193.107.17.72 port 29249 ssh2
May 18 12:26:01 webserver sshd[8546]: Failed password for invalid user root from 193.107.17.72 port 29316 ssh2
May 18 12:26:01 webserver sshd[8549]: Failed password for invalid user root from 193.107.17.72 port 29455 ssh2
May 18 12:26:01 webserver sshd[8548]: Failed password for invalid user root from 193.107.17.72 port 29382 ssh2
May 18 12:26:02 webserver sshd[8545]: Failed password for invalid user root from 193.107.17.72 port 29249 ssh2
May 18 12:26:02 webserver sshd[8553]: Failed password for invalid user root from 193.107.17.72 port 29580 ssh2
May 18 12:26:02 webserver sshd[8546]: Failed password for invalid user root from 193.107.17.72 port 29316 ssh2
May 18 12:26:02 webserver sshd[8548]: Failed password for invalid user root from 193.107.17.72 port 29382 ssh2
May 18 12:26:02 webserver sshd[8549]: Failed password for invalid user root from 193.107.17.72 port 29455 ssh2
May 18 12:26:02 webserver sshd[8545]: Failed password for invalid user root from 193.107.17.72 port 29249 ssh2
May 18 12:26:02 webserver sshd[8553]: Failed password for invalid user root from 193.107.17.72 port 29580 ssh2
May 18 12:26:02 webserver sshd[8546]: Failed password for invalid user root from 193.107.17.72 port 29316 ssh2
May 18 12:26:02 webserver sshd[8553]: Failed password for invalid user root from 193.107.17.72 port 29580 ssh2
May 18 12:26:02 webserver sshd[8548]: Failed password for invalid user root from 193.107.17.72 port 29382 ssh2
May 18 12:26:02 webserver sshd[8549]: Failed password for invalid user root from 193.107.17.72 port 29455 ssh2
May 18 12:26:03 webserver sshd[8553]: Failed password for invalid user root from 193.107.17.72 port 29580 ssh2
May 18 12:26:03 webserver sshd[8548]: Failed password for invalid user root from 193.107.17.72 port 29382 ssh2
May 18 12:26:03 webserver sshd[8548]: Failed password for invalid user root from 193.107.17.72 port 29382 ssh2
May 18 12:26:03 webserver sshd[8553]: Failed password for invalid user root from 193.107.17.72 port 29580 ssh2
May 18 12:26:04 webserver sshd[8553]: Failed password for invalid user root from 193.107.17.72 port 29580 ssh2
May 18 12:26:04 webserver sshd[8548]: Failed password for invalid user root from 193.107.17.72 port 29382 ssh2
May 18 12:26:04 webserver sshd[8869]: Failed password for invalid user root from 193.107.17.72 port 29760 ssh2

Open in new window

Interestingly, my tcpdump log has only 8 entries for this IP, 7 of which are *before* the 1st log message in /var/log/message, and only one for the 12:26 minute interval. That's puzzling. I am filitering the tcpdump on SYN and port 22, so perhaps not all of the login attempts were SYN?
2014-05-18 12:20:27.889262 IP 193.107.17.72.12870 > 64.129.23.95.22: Flags [S], seq 3115571362, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 266106506 ecr 0], length 0
2014-05-18 12:25:57.200499 IP 193.107.17.72.29249 > 64.129.23.95.22: Flags [S], seq 3926577031, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 266435809 ecr 0], length 0
2014-05-18 12:25:57.336530 IP 193.107.17.72.29316 > 64.129.23.95.22: Flags [S], seq 2026192561, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 266435946 ecr 0], length 0
2014-05-18 12:25:57.404111 IP 193.107.17.72.29382 > 64.129.23.95.22: Flags [S], seq 2442083427, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 266436011 ecr 0], length 0
2014-05-18 12:25:57.465870 IP 193.107.17.72.29455 > 64.129.23.95.22: Flags [S], seq 1500196181, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 266436073 ecr 0], length 0
2014-05-18 12:25:59.703992 IP 193.107.17.72.29580 > 64.129.23.95.22: Flags [S], seq 4137378115, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 266438313 ecr 0], length 0
2014-05-18 12:26:03.265366 IP 193.107.17.72.29760 > 64.129.23.95.22: Flags [S], seq 3154324675, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 266441874 ecr 0], length 0

Open in new window

I let this iptables configuration run all day.

Using the log configuration I had:

iptables -A INPUT -i eth0 -p tcp -j LOG --log-level 6 --log-prefix "iptables Accepted "

I logged 729 messages between  12:27PM and 00:18 the next day (today). I doubt it would be interesting to post them all, but here are the last 5:
May 19 00:12:19 webserver kernel: [1110207.624626] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=101.226.167.211 DST=64.129.23.95 LEN=40 TOS=0x08 PREC=0x20 TTL=42 ID=0 DF PROTO=TCP SPT=3927 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
May 19 00:12:19 webserver kernel: [1110207.624654] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=101.226.167.211 DST=64.129.23.95 LEN=40 TOS=0x08 PREC=0x20 TTL=42 ID=0 DF PROTO=TCP SPT=3927 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
May 19 00:12:21 webserver kernel: [1110210.224173] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=101.226.167.209 DST=64.129.23.95 LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=0 DF PROTO=TCP SPT=9045 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
May 19 00:12:21 webserver kernel: [1110210.224192] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=101.226.167.209 DST=64.129.23.95 LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=0 DF PROTO=TCP SPT=9045 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
May 19 00:18:37 webserver kernel: [1110585.515556] iptables Accepted IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=218.66.170.177 DST=64.129.23.95 LEN=40 TOS=0x00 PREC=0x00 TTL=106 ID=256 PROTO=TCP SPT=6000 DPT=222 WINDOW=16384 RES=0x00 SYN URGP=0

Open in new window

I have no /var/log/messages entries for the "SSH Break-in attempt" LOG/DROP rules.

Regarding various of your questions/comments:

> Your LOG rule looks all right to my sleepy eyes but are you only interested in TCP? I had a UDP DOS attack recently (to the NTP port, which is accepted and routed).

Yes, I think I'm only interested in TCP. I am not (yet) using ntp, still using netdate, mostly outgoing, but I have an accept rule for host 64.129.23.99 which synchronizes its time this this host. 37 is TCP. I removed my UDP port 53 and since my INPUT chain defaults to DROP, I think I am otherwise protected from any other UDP, yes?

> Also, are you running these firewall rules on the actual server system? It's recommended to have a router / firewall dedicated system with the servers on a protected LAN behind it.

Kind of long story, but the office Domain LAN is on the same 192.160.0/16 subnet, but the Domain controller has a separate public IP. This host does the website and some java jobs and I've configured it to have its own firewall. The firewall works really well, but I want to beef it up by limiting port 22 failed accesses - hence this EE Question.

> I would stick with the INPUT default ACCEPT and drop packets after logging them if you want. You know what's getting dropped that way

Hmmm, if I ACCEPT all, why wouldn't that take precedence over subsequent DROPs? How would I then prevent connection to e.g. port 139, or any of the other verboten ports w/o explicit DROPs in iptables? Anyway, the default of DROP for ACCEPT does not appear to be hurting anything once I reinstated:

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

> N.B. policy DROP will drop UDP and ICMP, which could be the root cause. You really should do something about these protocols, if you are going to persist with policy DROP (which I do not recommend).

Please illuminate me further on this recommendation. I don't think I need UDP or ICMP anyway, but given my concern in the previous point about letting anything through, why exactly do you not recommend default DROP for INPUT? The fellow who originally recommended this was all adamant about it.

> Also your LOG line should be:  -A INPUT -i eth0 -j LOG --log-level 6 --log-prefix "iptables drop-through "

OK, I'll make that change now and we'll see what happens.

Meanwhile, I'm tcpdump'ing on port 22 and SYN only. Do you have a recommendation for expanding that?

Author

Commented:
OK, I shut down that new log message you gave me. It logged 685 messages in the first 22 minutes. Below are  5 of the entries. All but 18 of them (1 shown below) are to DST=255.255.255.255. 12 of those 18 are from different IPs in the 69.191.249.0/24 range. 23 are specifically to DST=64.129.23.95 (this host's IP), 6 of which are from my remote ssh client.

Lots of SRC=0.0.0.0 DST=255.255.255.255, lots of PROTO=UDP (1122 of 1147 messages, so I guess UDPs aren't getting blocked, eh?).

All the PROTO=TCPs are to ports 80 or 443, so I suppose that's OK.

Here's the interesting bit to me, 325 of these accesses are from 11 different IP addresses in the 64.129.23.0/24 range. Our company has only 3 of these IPs. The rest are other occupants of the building!

Does this tell us anything? Should we weed out some of these (SRC=0.0.0.0, my workstation)?

Too much data to post, 1147 messages by the time I stopped the logging.
May 19 01:36:40 webserver kernel: [1115262.539909] iptables drop-through IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=69.191.249.202 DST=64.129.23.95 LEN=40 TOS=0x08 PREC=0x40 TTL=51 ID=0 DF PROTO=TCP SPT=37180 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
May 19 01:38:07 webserver kernel: [1115349.225525] iptables drop-through IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0c:42:e1:d4:56:08:00 SRC=64.129.23.121 DST=255.255.255.255 LEN=121 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=5678 DPT=5678 LEN=101
May 19 01:38:07 webserver kernel: [1115349.595394] iptables drop-through IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:d4:ca:6d:6e:d4:94:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=16 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
May 19 01:38:09 webserver kernel: [1115351.146205] iptables drop-through IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:d4:ca:6d:c3:f7:0c:08:00 SRC=64.129.23.110 DST=255.255.255.255 LEN=133 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=5678 DPT=5678 LEN=113
May 19 01:38:09 webserver kernel: [1115351.597180] iptables drop-through IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:d4:ca:6d:38:ad:20:08:00 SRC=64.129.23.122 DST=255.255.255.255 LEN=123 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=5678 DPT=5678 LEN=103

Open in new window

Author

Commented:
A bit more analysis: Of the 1147 messages logged, 12 were UDP messages not to DST=255.255.255.255. Of these, 8 were from itself (64.129.23.95) to port 138, DST=64.129.23.127. This DST is *not* one of our computers (unless our ISP hasn't told me something). UDP port 138 is NETBIOS Datagram Service (according to my research). This one has got me a bit worried.

3 of these UDP messages were 3 different sources to port 53 (DNS) on the local host. I removed the ACCEPT for UDP 53. Does it make sense that these would show up in this log message?

The final message in this group is from some remote host to the local host, UDP port 38497. I didn't find anything this port was specifically associated with, but neither this port nor 53 are enabled on this host.

I'm checking with the ISP on 64.129.23.127.

If we can filter out broadcast and possibly ports 80 and 443, this log message might prove useful. How would I do that? ...

iptables -A INPUT -i eth0  ! -d 255.255.255.255 -j LOG --log-level 6 --log-prefix "iptables drop-through "

The manpage says, "A "!"  argument before the address specification inverts the sense of the address", but the manpage itself shows the ! before the "-d". Confusing.

In fact, I'm trying my propsed  syntax. It didn't give an error and, so far, 1 log message. I'll let it run overnight. Listing the rule gives:
119      1    60 LOG        all  --  eth0   *       0.0.0.0/0           !255.255.255.255     LOG flags 0 level 6
 prefix `iptables drop-through '

Open in new window

Duncan RoeSoftware Developer

Commented:
Whether they are ACCEPTed or DROPped depends on the chain policy: you are just logging them.
Duncan RoeSoftware Developer

Commented:
I don't think I need UDP or ICMP anyway... without ICMP your system won't answer pings. That may or may not matter in your special situation, but some software systems use a ping to determine whether a host is alive and make no further attempt to contact that host on no ping response.
UDP is used by DNS at least.
Duncan RoeSoftware Developer

Commented:
Please illuminate me further on this recommendation ... the fellow who originally recommended this was all adamant about it.
Last one for tonight: I meant that the last rule in your INPUT chain could be iptables -A INPUT -i eth0 -j DROP. That gives you an effective DROP policy for eth0. But if the chain has policy ACCEPT, lo and eth1 packets will just go through without the need for any rules at all. (Although you might want to ACCEPT them early to speed things up, since you have more than 100 rules).

Author

Commented:
>  some software systems use a ping to determine whether a host is alive and make no further attempt to contact that host on no ping response.

Long ago I disabled ping by other means. At the time it was to block DOS attacks via pinging. For our purposes, I think having a host make no further attempts to contact is a good thing. I don't think we want external hosts to determine if we are alive.

 > UDP is used by DNS at least.

The local DNS server is the SBS Domain server on the local LAN (eth1), which is ACCEPT across the board. However, even if we were using an external DNS, wouldn't this need to be for OUTPUT only? Wouldn't we only need UDP 53 INPUT if we were letting outside servers use this host for DNS resolution? Our UDP OUTPUT is not blocked.

>   the last rule in your INPUT chain could be iptables -A INPUT -i eth0 -j DROP. That gives you an effective DROP policy for eth0.
>  But if the chain has policy ACCEPT, lo and eth1 packets will just go through without the need for any rules at all. (Although you > might want to ACCEPT them early to speed things up, since you have more than 100 rules).

The ACCEPT rules for eth0 and lo are rules 3 and 4, after the RELATED,ESTABLISHED rule and before all the other rules.

In fact, there are only 19 actual rules, including your chain creation rules and including the "iptables drop-through" log. All the others are ad hoc INPUT DROPs for specific IPs determined to have been trying to break in, DOS, etc. via ssh, ftp, mail, or the website. These are cleared out periodically so as not to grow infinitely. I end up maxing out at about 120+ blocks before they all get remove.

Still looking for that silver bullet rule to get --limit and --hitcount working. Still nothing logged with the "SSH Break-in attempt" log rule.

Over the past 8+ hours I've logged 427 "iptables drop-through" messages (after eliminating DST=255.255.255.255). I'll do some analysis on this and post the results. Hopefully this gives some insight into the --hitcount issue?

Author

Commented:
Analysis of "iptables drop-through" log.

427 Total messages logged between 3:01AM and 11:46AM (excluding DST=255.255.255.255)

301 for ports 443,80 - Since we're hosting 2 websites, I think these are OK.

10 of the remaining 126 are UDP 53, 6 from the local host to the outside (none to locally defined nameservers), 4 from the outside to this host. These are a bit of a puzzle, but I don't think are an issue?

33 of the UDP messages are mostly to high-numbered ports. No idea what is going on there. Two examples:
May 19 11:31:22 webserver kernel: [1150900.754377] iptables drop-through IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=193.110.157.2 DST=64.129.23.95 LEN=76 TOS=0x00 PREC=0x00 TTL=46 ID=47261 PROTO=UDP SPT=53 DPT=38497 LEN=56
May 19 11:34:29 webserver kernel: [1151087.470348] iptables drop-through IN=eth0 OUT= MAC=00:c0:a8:7b:93:1f:d4:ca:6d:6e:d4:94:08:00 SRC=69.58.113.104 DST=192.168.0.3 LEN=122 TOS=0x00 PREC=0x00 TTL=63 ID=32029 PROTO=UDP SPT=25269 DPT=61112 LEN=102

Open in new window

However, there is one request to port 17 (qotd - quote of the day!?) and one is to port 137 (netbios-ns - NetBIOS name service). Not sure what these are about, but only 1 each, so no big deal? These ports are not opened.

88 of the UDP messages are SRC=64.129.23.95 DST=64.129.23.127 PROTO=UDP DPT=138. I checked with our ISP and 64.129.33.127 is not an assigned IP in this subnet. I'm curious/concerned about this one, but perhaps out of the scope of this question's objective. I'll post this elsewhere.

There are 32 non 80/443 TCP messages

3 of these are to port 808x which is obviously someone trying to remote-log into a router/firewall.
 
3 are to port 21 (ftp) and 3 are to port 23 (telnet) neither of which service is enabled.

4 to port 3389 (Microsoft remote desktop), not enabled or available on linux

3 to port 1433 - the normal SQL Server port - not enabled or available

The rest are for various high-numbered ports: 1688 (1) [MS Key Management Service],  2083 (1), 21243 (1), 22799 (1), 26380 (1), 2832 (1), 502 (1) [asa-appl-proto), 5038 (1), 5631 (1) [pc-anywhere], 56754 (1), 56957 (1), 5900 (1) [vnc and apparently various viruses], 6217 (1).

Do these log results give you any insight into our --limit/--hitcount issue? They don't tell me much.

Author

Commented:
Still no "SSH Break-in attempt" logs.

OK, I expect the "iptables drop-through" log messages aren't going to help us figure out what's wrong with the --limit and --hitcount. I'm running iptables 1.4.10. I'm going to go ahead and upgrade to 1.4.21 and see if there is something that has been fixed since my version. I'll let this run and post back results.

Meanwhile, please interpret for me your checkcount chain logic:
iptables -N checkcount
iptables -A checkcount -m recent ! --update --seconds 60 -j setaddr
iptables -A checkcount -m recent --rcheck --hitcount 10 -j logdrop
iptables -A checkcount -j RETURN

Open in new window

The rule on line 2 says what, ... match IP against recent list seen within last 60 seconds, if not in the list  join the setaddr chain?

line 3: check if the IP is in the recent list and 10 packets have been received. If so, join the logdrop chain?

I'm doubtful I'm interpreting that right. Tell me what it really means. I'd like to see if I can see some reason why I'm not getting any matches.

The setaddr chain rules simply sets IP into the recent list, right?
Duncan RoeSoftware Developer

Commented:
Lots of interesting stuff there, may be input for another EE Q or 2.
I agree with you - we need to put these aside and concentrate on why hit_count isn't doing what you want.
From reading the source, it does look to me like we have to issue a set for every received matching packet. The man page does say If the source address is already in the list, this (set) will update the existing entry. In this context, updating includes counting the packet. At least, that's my take: you might like to cast your eye over /usr/src/linux/net/netfilter/xt_recent.c for a second opinion. This is what I would try
iptables -A checkcount -m recent --set
iptables -A checkcount -m recent --rcheck --hitcount 10 -j logdrop

Open in new window

Chain setaddr is no longer required.

Author

Commented:
Well, I've confirmed that it is not the iptables version. Within an hour of using the new version I received 98 ssh attempts and 22 ssh attempts. :(

OK, so the new script will be (minus "initialization"):
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --syn -j ACCEPT

iptables -N logdrop
iptables -A logdrop -j LOG --log-level 6 --log-prefix "SSH Break-in attempt "
iptables -A logdrop -j DROP

iptables -N checkcount
iptables -A checkcount -m recent --set
iptables -A checkcount -m recent --rcheck --hitcount 10 -j logdrop
iptables -A checkcount -j RETURN

iptables -A INPUT -p tcp --syn --dport 22 -i eth0 -j checkcount
iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -j ACCEPT

iptables -A INPUT -i eth0 -p tcp --syn -m multiport --dports 25,53,80,443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 -m multiport --dports 22,37 -j ACCEPT

Open in new window


I'll give that a shot and post back results.

Author

Commented:
Possibly some progress! I've finally logged some "SSH Break-in attempt" messages: 10 between 5:54AM and 21:54PM. Here are the interesting fields from the log message (I can post the whole messages if you want):
May 20 05:54:19 SRC=58.222.1.154 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 05:54:22 SRC=58.222.1.154 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 05:54:28 SRC=58.222.1.154 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 09:52:41 SRC=202.101.173.109 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 09:52:44 SRC=202.101.173.109 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 09:52:50 SRC=202.101.173.109 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 19:11:31 SRC=58.222.1.154 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 19:11:34 SRC=58.222.1.154 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 19:11:40 SRC=58.222.1.154 DST=64.129.23.95 PROTO=TCP DPT=22
May 20 21:54:59 SRC=65.111.173.41 DST=64.129.23.95 PROTO=TCP DPT=22

Open in new window

Some observations - None of these offenders appear to have logged either these "SHH Break-in" messages, or "Failed password" messages within the same second, so perhaps the --limit is working (!)

There are 6 log messages for IP 58.222.1.154; 3 at 5:54 and 3 at 19:11.

I do have 9 "Failed password" login attempts for this IP prior to the first "SSH break-in" log message at 5:54:19, so that indicates to me that the 10th one triggered the log. Do you concur?

No "Failed password" messages were logged after the 3 "SSH Break-in" message starting at 5:54:19 which must mean the IP was DROPped by the logdrop chain. Why, I wonder, would there be 2 additional log messages after the DROP? Would this have to do with the burst limit?

I'll have to change this --hitcount to more than 10 since it apparently blocks on the 10th. Therefore, only 9 "Failed password" messages get logged. My cron script looks for 10 or more of these "Failed password" message and, if found, adds a DROP rule for that IP. SInce only 9 were found no DROP rule was created. Therefore, this IP was able to try again at 19:11. Interestingly, there are no "Failed password" messages from 19:11 -- not sure why.

I'm going to change the --hitcount to 12 and see if that lets 10 (11 actually) "Failed password" messages get logged.

Author

Commented:
Actually, not sure about --limit working. Here's an example with 2 attempts within the same second:
May 20 16:00:00 webserver sshd[10603]: Invalid user admin from 124.40.250.115
May 20 16:00:00 webserver sshd[10601]: Failed password for invalid user root from 124.40.250.115 port 7176 ssh2
May 20 16:00:00 webserver sshd[10603]: Failed password for invalid user admin from 124.40.250.115 port 41538 ssh2

Open in new window

However, that's the only one I've found so far so maybe the attacker snuck 2 requests for different users in in the 1st second, but was possibly nipped in the bud after that. I wasn't logging tcpdump at the time. Am doing so now.
Duncan RoeSoftware Developer
Commented:
Looks to me like it's working.
You could reduce the count in the cron job ... and the hit count (which has to stay 1 more than the cron job)

Author

Commented:
> Looks to me like it's working.

Waiting for the next attack to confirm ... but I think you're right!
Commented:
Well, I think we've got it! All attacks get nipped at 10 attempts. This was confirmed when I accidently forget to remove my firewall reset logic from cron and took down the firewall - bam! 158 attempts. Below is my final script, including initialization.

Thanks for your help on this. I've been trying to find a solution for month and have posted this problem on multiple site. No one was able to figure it out.
    /usr/sbin/iptables -P INPUT DROP
    /usr/sbin/iptables -P FORWARD DROP

    /usr/sbin/iptables -t nat -P PREROUTING ACCEPT
    /usr/sbin/iptables -t nat -P POSTROUTING ACCEPT

    /usr/sbin/iptables -t mangle -P PREROUTING ACCEPT
    /usr/sbin/iptables -t mangle -P INPUT ACCEPT
    /usr/sbin/iptables -t mangle -P FORWARD ACCEPT
    /usr/sbin/iptables -t mangle -P POSTROUTING ACCEPT

    /usr/sbin/iptables -t raw -P PREROUTING ACCEPT

    /usr/sbin/iptables -F
    /usr/sbin/iptables -F -t nat
    /usr/sbin/iptables -F -t mangle
    /usr/sbin/iptables -F -t raw

    /usr/sbin/iptables -X
    /usr/sbin/iptables -X -t nat
    /usr/sbin/iptables -X -t mangle
    /usr/sbin/iptables -X -t raw

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --syn -j ACCEPT

iptables -N logdrop
iptables -A logdrop -j LOG --log-level 6 --log-prefix "SSH Break-in attempt "
iptables -A logdrop -j DROP

iptables -N checkcount
iptables -A checkcount -m recent --set
iptables -A checkcount -m recent --rcheck --hitcount 12 -j logdrop
iptables -A checkcount -j RETURN

iptables -A INPUT -p tcp --syn --dport 22 -i eth0 -j checkcount
iptables -A INPUT -p tcp --syn -m limit --limit 1/s -i eth0 --dport 22 -j ACCEPT

iptables -A INPUT -i eth0 -p tcp --syn -m multiport --dports 25,53,80,443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp -s 64.129.23.99 -m multiport --dports 22,37 -j ACCEPT

Open in new window

Author

Commented:
My final posting contains the consolidated solution.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial