Solved

iptables not blocking IP, wrong specification?

Posted on 2014-03-14
25
928 Views
Last Modified: 2014-03-19
I have a pretty big problem that's happening right now. I have a hacker trying to break into my Linux server. There are currently over 2500 attempt logged in /var/log/messages as:

Mar 15 01:06:44 webserver sshd[32160]: reverse mapping checking getaddrinfo for ip197.hichina.com [121.197.8.22] failed - POSSIBLE BREAK-IN ATTEMPT!
Mar 15 01:06:44 webserver sshd[32160]: Invalid user ftpuser2 from 121.197.8.22
Mar 15 01:06:44 webserver sshd[32160]: Failed password for invalid user ftpuser2 from 121.197.8.22 port 57361 ssh2

Open in new window


The really odd thing is that I have a script to detect these things and if there are more than 10 attempts it blocks the IP using iptables. In fact, when I run: iptables -L -v, I do get:

    0     0 DROP       all  --  any    any     121.197.0.0/16       anywhere

Open in new window


This entry was created at 2014-03-15 00:39, yet you can see that the example entries are more than 1/2 an hour later ... and attempts continue to be made; about 200 attempts every 6 minutes.

Why is iptables not blocking this IP? Am I specifying the address wrong? Also, why are they able to try on port 57361? I don't have that port open:

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p TCP -i eth0 -m multiport --dports 22,25,53,80,443 -m state --state NEW -j ACCEPT
iptables -A INPUT -p UDP -i eth0 -m multiport --dports 53 -m state --state NEW -j ACCEPT

Open in new window


This is rather urgent. I'm under attack NOW!
0
Comment
Question by:jmarkfoley
  • 15
  • 7
  • 3
25 Comments
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39930962
Note, when I run: iptables -A INPUT -s 121.197.8.22 -j DROP, by hand the entry goes in as:
0     0 DROP       all  --  any    any     ip197.hichina.com    anywhere

Open in new window

but still no blocking. The hacker keeps trying. 5183 attempts as of this posting.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39930967
when I try ssh'ing to this port from an external computer, it just sits there for 3 minutes doing nothing, then gives me a "Connection timed out", whether or not I use a legitimate ID. I never get a password prompt. How is this guy getting "Failed password for user www ..." 6 times per minute on completely bogus ports? 6167 attempts ... and counting ...
0
 
LVL 27

Expert Comment

by:serialband
ID: 39931415
Install faill2ban or denyhosts to block those for you automatically.

That log entry is standard brute force attacks on your open ssh ports.  It's from several automated sources.  You could also change you ssh port to something above port 1024 and that would stop all the automated attempts.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39931503
I'll be glad to look at other software as a final option, but why is iptables not blocking those IPs?

Can't really change the ssh port, but why should those ports be open in the first place? My complete iptables config is below. I believe these ports are not opened.
IPT="/usr/sbin/iptables"

if [ -z "$1" ] || [ "$1" = "start" ]
then
    echo Starting Firewall
    $IPT -P INPUT DROP
    $IPT -P FORWARD DROP
    $IPT -P OUTPUT DROP

    $IPT -t nat -P PREROUTING ACCEPT
    $IPT -t nat -P POSTROUTING ACCEPT
    $IPT -t nat -P OUTPUT ACCEPT

    $IPT -t mangle -P PREROUTING ACCEPT
    $IPT -t mangle -P INPUT ACCEPT
    $IPT -t mangle -P FORWARD ACCEPT
    $IPT -t mangle -P OUTPUT ACCEPT
    $IPT -t mangle -P POSTROUTING ACCEPT

    $IPT -t raw -P PREROUTING ACCEPT
    $IPT -t raw -P OUTPUT ACCEPT

    $IPT -F
    $IPT -F -t nat
    $IPT -F -t mangle
    $IPT -F -t raw

    $IPT -X
    $IPT -X -t nat
    $IPT -X -t mangle
    $IPT -X -t raw

    $IPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
    $IPT -A INPUT -i lo -j ACCEPT
    $IPT -A INPUT -i eth1 -m state --state NEW -j ACCEPT

    $IPT -A INPUT -p TCP -i eth0 -m multiport  --dports 22,25,53,80,443 -m state --state NEW -j ACCEPT

    $IPT -A INPUT -p UDP -i eth0 -m multiport --dports 53 -m state --state NEW -j ACCEPT
    $IPT -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
    $IPT -I INPUT -p tcp -m tcp -s 64.129.23.99 --dport 37 -j ACCEPT
    $IPT -A OUTPUT -o eth0 -m state --state NEW -j ACCEPT
    $IPT -A OUTPUT -o lo -j ACCEPT
    $IPT -A OUTPUT -o eth1 -m state --state NEW -j ACCEPT

Open in new window

0
 
LVL 27

Expert Comment

by:serialband
ID: 39931586
Line 37 accepts all IP addresses for port 22 as well as the other ports you've entered.  I don't see any drop or reject rules in your list.

/sbin/iptables -A INPUT -s 1.2.3.4 -j DROP
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39931834
Yes, it does initially accept all connections to port 22. I do have 89 drop rules all of which  I didn't show, but I did show the ones for the example IP in my initial post and in my post ID: 39930962:

$ iptables -A INPUT -s 121.197.0.0/16 -j DROP
$ iptables -A INPUT -s 121.197.8.22 -j DROP
$iptables -L -v

0     0 DROP       all  --  any    any     121.197.0.0/16       anywhere
0     0 DROP       all  --  any    any     ip197.hichina.com    anywhere

Secondly, As I mentioned, these are apparently not attempts on port 22 but on random higher numbered ports as shown in the example /var/log/messages output in my initial post: port 57361.

So the questions remain: why are my DROPs for IPs 121.197.0.0/16 and 121.197.8.22 not blocking this IP for ALL ports, and why is the attempt getting through on port 57361 when that port is not enabled?
0
 
LVL 76

Accepted Solution

by:
arnold earned 275 total points
ID: 39932087
iptables -L INPUT --line-numbers

note that when you use the -A it adds the new entry at the end, in the case such as yours, you want to use iptables -t filter -I INPUT 1 -s 121.197.0.0/16 -j DROP
 to add the entry near at the top.

in your case, you have an allow rule to ssh such that the drop is never consulted,

IPTABLES rules are evealuated from the top down in order.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39932220
Sounds like you might be on to something. I'll change my DROPs to be as you've shown and check it out. In fact, iptables -L -v --line-numbers does show the DROPs in there first in reverse order.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39932223
While I'm waiting for an intruder ... what's the deal with allowing anything at all on port 57361 and the like? I though my INPUT rules on eth0 permitted only ports 22, 25, 53, 80, 443
0
 
LVL 76

Expert Comment

by:arnold
ID: 39932372
Without seeing what your iptables reflect, I am unclear where or what that reference means or is coming from.

Post if you can your iptables -t filter -L --line-numbers
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39932503
I guess the port number on the /var/log/message entry isn't the port number it's trying to connect to. It must be the internal port number it assigns. A legitimate connection to port 22 also give a large port number:

Mar 15 01:06:44 webserver sshd[32160]: Failed password for invalid user ftpuser2 from 121.197.8.22 port 57361 ssh2

So, I think my question in post ID: 39932223 is likely nonsense.

Here is the iptables -t filter -L --line-numbers (with most of the DROPs removed):
> iptables -t filter -L --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       all  --  191.234.0.0/16       anywhere
2    DROP       all  --  211.70.0.0/16        anywhere
3    DROP       all  --  199-127-56.static.fiberhub.net/21  anywhere
4    DROP       all  --  ip197.hichina.com    anywhere
5    DROP       all  --  121.197.0.0/16       anywhere
:
91   ACCEPT     tcp  --  mail.[redacted user domain]   anywhere            tcp dpt:time
92   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
93   ACCEPT     all  --  anywhere             anywhere
94   ACCEPT     all  --  anywhere             anywhere            state NEW
95   ACCEPT     tcp  --  anywhere             anywhere            multiport dports ssh,smtp,domain,http,https state NEW
96   ACCEPT     udp  --  anywhere             anywhere            multiport dports domain state NEW

Chain FORWARD (policy DROP)
num  target     prot opt source               destination

Chain OUTPUT (policy DROP)
num  target     prot opt source               destination
1    ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
2    ACCEPT     all  --  anywhere             anywhere            state NEW
3    ACCEPT     all  --  anywhere             anywhere
4    ACCEPT     all  --  anywhere             anywhere            state NEW

Open in new window

0
 
LVL 76

Expert Comment

by:arnold
ID: 39932647
note your 91-94 accept all connections.

You could try using instead of -DROP -logdrop and then you can see whether the events are caught and reflected.

You might be better to setup a CHAIN

deny_these_people.

add this chain at the top of your INPUT table.

Then you would add the deny drop rules into the deny_these_people chain.

iptables -t filter -N deny_these_people
iptables -t filter -I INPUT 1 -j deny_these_people

test on one event first using -j log within the chain to make sure it works as you expect.

Only define items you want blocked in the chain.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 1

Author Comment

by:jmarkfoley
ID: 39932963
> note your 91-94 accept all connections.

I don't see where this is set. Below is my script to initialize iptables which line is allowing 91-94 accept all connections? I need to shut that down.

>You could try using instead of -DROP -logdrop and then you can see whether the events are caught and reflected.

Looks interesting. Would that syntax be:

iptables -t filter -I INPUT 1 -s 121.197.0.0/16 -j logdrop

> You might be better to setup a CHAIN
> deny_these_people.
> add this chain at the top of your INPUT table.
> Then you would add the deny drop rules into the deny_these_people chain
>
> iptables -t filter -N deny_these_people
> iptables -t filter -I INPUT 1 -j deny_these_people

You just lost me! What is "deny_these_people"? A file containing IP addresses? I've never (knowingly) done chains before. According to the manpage, it is the name of the new chain (so probably not a file). How would I add the rules to the chain?

rc.firewall script (DROP rules are sourced)
     1
     2  IPT="/usr/sbin/iptables"
     3
     4  if [ -z "$1" ] || [ "$1" = "start" ]
     5  then
     6      echo Starting Firewall
     7      $IPT -P INPUT DROP
     8      $IPT -P FORWARD DROP
     9      $IPT -P OUTPUT DROP
    10
    11      $IPT -t nat -P PREROUTING ACCEPT
    12      $IPT -t nat -P POSTROUTING ACCEPT
    13      $IPT -t nat -P OUTPUT ACCEPT
    14
    15      $IPT -t mangle -P PREROUTING ACCEPT
    16      $IPT -t mangle -P INPUT ACCEPT
    17      $IPT -t mangle -P FORWARD ACCEPT
    18      $IPT -t mangle -P OUTPUT ACCEPT
    19      $IPT -t mangle -P POSTROUTING ACCEPT
    20
    21      $IPT -t raw -P PREROUTING ACCEPT
    22      $IPT -t raw -P OUTPUT ACCEPT
    23
    24      $IPT -F
    25      $IPT -F -t nat
    26      $IPT -F -t mangle
    27      $IPT -F -t raw
    28
    29      $IPT -X
    30      $IPT -X -t nat
    31      $IPT -X -t mangle
    32      $IPT -X -t raw
    33
    34      x=`lsmod | grep conntrack_ftp`
    35
    36      $IPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
    37
    38      $IPT -A INPUT -i lo -j ACCEPT
    39
    40      $IPT -A INPUT -i eth1 -m state --state NEW -j ACCEPT
    41
    42      $IPT -A INPUT -p TCP -i eth0 -m multiport \
    43          --dports 22,25,53,80,443 -m state --state NEW -j ACCEPT
    44
    45      $IPT -A INPUT -p UDP -i eth0 -m multiport --dports 53 -m state --state NEW -j ACCEPT
    46
    47      $IPT -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
    48
    49      $IPT -I INPUT -p tcp -m tcp -s 64.129.23.99 --dport 37 -j ACCEPT
    50
    51      $IPT -A OUTPUT -o eth0 -m state --state NEW -j ACCEPT
    52
    53      $IPT -A OUTPUT -o lo -j ACCEPT
    54
    55      $IPT -A OUTPUT -o eth1 -m state --state NEW -j ACCEPT
    56
    57  . /etc/intrude/blockIP
    58
    59  elif [ "$1" = "stop" ]
    60  then
   :
    90  fi

Open in new window

0
 
LVL 27

Assisted Solution

by:serialband
serialband earned 225 total points
ID: 39932973
You could rate limit your port 22 connections.  It's not really necessary to block each one individually.

iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent \
  --update --seconds 60 --hitcount 4 -j DROP
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39933169
I read the manpage on hitcount and don't quite understand. In English, what does your example do? Restrict an ssh attempt from an IP for 60 seconds after 4 failed connections?
 
Also, could you explain what the logdrop option does and what the correct syntax is? My proposed syntax:

iptables -t filter -I INPUT 1 -s 121.197.0.0/16 -j logdrop

doesn't work and I get the message:

iptables -t filter -I INPUT 1 -s 162.13.0.0/16 -j logdrop
iptables v1.4.10: Couldn't load target `logdrop':/usr/libexec/xtables/libipt_logdrop.so:

I did have another attack a little while ago - on two hosts concurrently. One host is using your recommended: iptables -t filter -I INPUT 1 -s 162.13.0.0/16 -j DROP, the other is using my prevous:  iptables -A INPUT -s 162.13.0.0/16 -j DROP, syntax. There were 306 attempts on each host. Furthermore, the attack persisted for 1:23 minutes after the DROP entry was created. This makes me feel like the DROP is not working at all, even with the newer syntax.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39933241
Skipped several/many steps.
logdrop
log
logaccept are usually created as additional chains with specific options

logdrop is supposed to log the events to syslog while droping the packet as the explicit rule within.

Not sure how you start, but the output you provided to iptables -t filter -L --line-numbers has lines 91-94 that allow all/everything.
.
.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39934540
OK, I get what you're saying on the logging. Not sure I need the additional log info since I can see the events in /var/log/messages.

> iptables -t filter -L --line-numbers has lines 91-94 that allow all/everything.

Yes, you are right. I do need to be able to access this host from anywhere since I administer it from offsite. However, I do have the user IDs restricted in /etc/ssh/sshd_config.

I think perhaps moving the DROP to above this ACCEPT part of the script should prevent the allow everything condition, right? I've got that configured right now and am waiting for the next attack to see if that does the trick.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39935093
Allowing access from anywhere to anything is what is listed in the four lines.

You can allow access to SSH.
Depending on your setup, you should consider configuring PPTP or other VPN type setup on your server.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39936013
> Allowing access from anywhere to anything is what is listed in the four lines.

Line 94 of the iptables -t filter -L --line-numbers list (post ID: 39932503) does accept all from anywhere, the others have specific ports. However, this listing does not show the interfaces. I believe that line corresponds to line 40 in my firewall script (post ID: 39932963):

iptables -A INPUT -i eth1 -m state --state NEW -j ACCEPT

Which only accepts for interface eth1. This is the internal LAN, not eth0 which is Internet facing. Am I interpreting that correctly?

Anyway, I think we're closing in on a solution (but please also confirm my statement above). Since putting the DROPs above the ACCEPTs I have not had an attack of more than 10 attempts spanning more than one minute.

I have, however, had attacks in the hundreds of attempts within one minute (so far max of 306 attempts within 1 minute), so I think that hitcount idea could solve that problem. How would I block for 1 minute 10 or more failed attempts? That would give my script the time it needs to catch and block the bugger. Here's my guess based on serialband's example in post ID: 39932973

iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent \
  --update --seconds 60 --hitcount 10 -j DROP

If that is correct, where would I put it? Above all INPUT ACCEPTs?

I'd like to give that a shot and maybe life will be good!
0
 
LVL 76

Expert Comment

by:arnold
ID: 39936078
You would need to add a number to INPUT line_number from the example above
You need only place it right above the allow port 22 access that applies to the external interface.

In a setup such as yours, I think you should look into using another chain

That when you add to the main INPUT
iptables -I INPUT 5 -i eth1 -j my_chain_for_eth0_rules
iptables -I INPUT 6 -i eth0 -j my_chain_for_eth0_rules


This way you have an easier way to manage as well as make sure that you do not mistakenly add a rule to the external interface that need only apply to the internal interface as may have happened here.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39937347
> You would need to add a number to INPUT line_number from the example above

Sorry, don't follow what you mean by this. Can you elaborate or provide an example?

> You need only place it right above the allow port 22 access that applies to the external interface.

So the --hitcount syntax is OK?

You're saying I should have it after my eth1 accept and before my eth0 accept (between lines 40 and 42 of my firewall script in post ID: 39932963) as follows:

iptables -A INPUT -i eth1 -m state --state NEW -j ACCEPT

iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent \
  --update --seconds 60 --hitcount 10 -j DROP

iptables -A INPUT -p TCP -i eth0 -m multiport \
        --dports 22,25,53,80,443 -m state --state NEW -j ACCEPT

Please confirm this looks correct and I'll stick it in the script immediately.

> In a setup such as yours, I think you should look into using another chain

You're probably right, but one change at a time. I'm no iptables guru so I need to move slowly. I'll try the additional chain idea later. It's not a problem if these IPs are block on all interfaces.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39938060
iptables -I INPUT 6 -p tcp --dport 22 -i eth0 -m state --state NEW -m recent \
  --update --seconds 60 --hitcount 10 -j DROP

iptables -I INPUT 7 -p TCP -i eth0 -m multiport \
        --dports 22,25,53,80,443 -m state --state NEW -j ACCEPT


note when you use the iptables -A INPUT option it appends the rule to the buttom of the existing list.  Using the iptables -I INPUT 2 means that this rule will be placed in position 2 of this table. This gives you a better mechanism of placing rules.  Deals with how many rules have to be evaluated before your required action is performed.

..
.
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39939694
OK, this appears to be doing the trick, although I don't appear to have had any big attacks for several days. I'll consider this on resolved for now and if I run into problems I'll repost something.
0
 
LVL 1

Author Closing Comment

by:jmarkfoley
ID: 39939710
Thanks for your help
0
 
LVL 1

Author Comment

by:jmarkfoley
ID: 39939875
Well, spoke too soon about everything working. It figures that no activity for several days, then an attack as soon as I close this question!

The DROP hitcount setting is not working. I've opened a new question:
http://www.experts-exchange.com/OS/Linux/Q_28392266.html
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now