fallonsupport
asked on
Linux Box Intrusion
Also posted to OS Linux but more points and more info posted here.
VA Linux 6.2.4, Internet Server runing Sendmail, DNS, radius, and HTTP/FTP services. Two identical
servers have been hacked by the same people.
ps aux >ps shows this command running
"./s3 0 200.255.64.103 1 56650 1 800"
The machine that has crashed was also showing these processes running.
1035 ? D 8:40 (nfsiod)
1036 ? R 7:57 (nfsiod)
2585 ? D 8:25 (nfsiod)
13011 ? Z 0:00 [osdbo <defunct>]
9964 ? D 6:58 (nfsiod)
19379 ? S 0:00 httpd
13296 ? D 7:39 (nfsiod)
9611 ? D 6:57 (nfsiod)
1245 ? D 6:58 (nfsiod)
5954 ? Z 0:00 [osdbo <defunct>]
1339 ? D 6:01 (nfsiod)
2475 ? D 6:04 (nfsiod)
31666 ? D 5:48 (nfsiod)
6106 ? D 5:36 (nfsiod)
23795 ? Z 0:00 [osdbo <defunct>]
28778 ? D 1:19 (nfsiod)
30060 ? D 1:05 (nfsiod)
27633 ? Z 0:00 [osdbo <defunct>]
10136 ? D 0:59 (nfsiod)
29318 ? D 0:49 (nfsiod)
18732 ? R 241:29 ./s3 0 200.215.129.6 1 56650 1
16670 ? D 0:16 (nfsiod)
31264 ? D 0:13 (nfsiod)
12316 ? S 0:00 (nfsiod)
7377 ? S 0:00 /usr/local/sbin/sshd
7420 ttyp0 S 0:00 -bash
29402 ? R 0:00 (nfsiod)
29403 ? R 0:00 (nfsiod)
29404 ? R 0:00 (nfsiod)
29405 ? R 0:00 (nfsiod)
29406 ? R 0:00 (nfsiod)
29407 ? R 0:00 (nfsiod)
29408 ? R 0:00 (nfsiod)
29409 ? R 0:00 (nfsiod)
29410 ? R 0:00 (nfsiod)
29411 ? R 0:00 (nfsiod)
29412 ? R 0:00 (nfsiod)
29413 ? R 0:00 (nfsiod)
29414 ? R 0:00 sh -c
29420 ? R 0:00 (nfsiod)
29421 ? R 0:00 (nfsiod)
29422 ? R 0:00 (nfsiod)
29423 ? R 0:00 (nfsiod)
The intruder apparently created the following folders:
/dev/. /s3 (that is a space after the period)
/dev/. /OSDBO
/dev/. /OSDBO2
The second machine where we have observed the
"./s3 0 200.255.64.103 1 56650 1 800" command
has not had the nfsiod processes showing.
There have been various IP addresses associated with this effort all so far from the .br domain.
The s3 folder appears to have the contents of the /dev directory. We rebooted one machine, and it's toast, would not boot back up. When the intruder attaches, we can disconnect him and he'll reattach almost immediately.
Does anybody recognize this hack process?
Can you me tell what it's doing, and most importantly how
to reverse the effects before this system crashes and prevent it from happening again?
VA Linux 6.2.4, Internet Server runing Sendmail, DNS, radius, and HTTP/FTP services. Two identical
servers have been hacked by the same people.
ps aux >ps shows this command running
"./s3 0 200.255.64.103 1 56650 1 800"
The machine that has crashed was also showing these processes running.
1035 ? D 8:40 (nfsiod)
1036 ? R 7:57 (nfsiod)
2585 ? D 8:25 (nfsiod)
13011 ? Z 0:00 [osdbo <defunct>]
9964 ? D 6:58 (nfsiod)
19379 ? S 0:00 httpd
13296 ? D 7:39 (nfsiod)
9611 ? D 6:57 (nfsiod)
1245 ? D 6:58 (nfsiod)
5954 ? Z 0:00 [osdbo <defunct>]
1339 ? D 6:01 (nfsiod)
2475 ? D 6:04 (nfsiod)
31666 ? D 5:48 (nfsiod)
6106 ? D 5:36 (nfsiod)
23795 ? Z 0:00 [osdbo <defunct>]
28778 ? D 1:19 (nfsiod)
30060 ? D 1:05 (nfsiod)
27633 ? Z 0:00 [osdbo <defunct>]
10136 ? D 0:59 (nfsiod)
29318 ? D 0:49 (nfsiod)
18732 ? R 241:29 ./s3 0 200.215.129.6 1 56650 1
16670 ? D 0:16 (nfsiod)
31264 ? D 0:13 (nfsiod)
12316 ? S 0:00 (nfsiod)
7377 ? S 0:00 /usr/local/sbin/sshd
7420 ttyp0 S 0:00 -bash
29402 ? R 0:00 (nfsiod)
29403 ? R 0:00 (nfsiod)
29404 ? R 0:00 (nfsiod)
29405 ? R 0:00 (nfsiod)
29406 ? R 0:00 (nfsiod)
29407 ? R 0:00 (nfsiod)
29408 ? R 0:00 (nfsiod)
29409 ? R 0:00 (nfsiod)
29410 ? R 0:00 (nfsiod)
29411 ? R 0:00 (nfsiod)
29412 ? R 0:00 (nfsiod)
29413 ? R 0:00 (nfsiod)
29414 ? R 0:00 sh -c
29420 ? R 0:00 (nfsiod)
29421 ? R 0:00 (nfsiod)
29422 ? R 0:00 (nfsiod)
29423 ? R 0:00 (nfsiod)
The intruder apparently created the following folders:
/dev/. /s3 (that is a space after the period)
/dev/. /OSDBO
/dev/. /OSDBO2
The second machine where we have observed the
"./s3 0 200.255.64.103 1 56650 1 800" command
has not had the nfsiod processes showing.
There have been various IP addresses associated with this effort all so far from the .br domain.
The s3 folder appears to have the contents of the /dev directory. We rebooted one machine, and it's toast, would not boot back up. When the intruder attaches, we can disconnect him and he'll reattach almost immediately.
Does anybody recognize this hack process?
Can you me tell what it's doing, and most importantly how
to reverse the effects before this system crashes and prevent it from happening again?
Somebody else may recognise the hack. In terms of what you need to do.
It is best security practice to always completely rebuild any compromised machine from a trusted backup or software distribution CDs. Unless you have real experts in house with time to burn you cannot be sure that a machine does not harbour dormant malicious code. The only time I would make an exception to this is with a documented virus/worm where there is a published clean up procedure or program.
To prevent a re-occurence you must harden your servers and keep up to date with security fixes. There are many guides around to do this. SANS have one of the better guides though does cost $49.
http://www.sansstore.org/ (look for the step by step security linux guide).
Other detailed information at:
http://www.linuxsecurity.com/
It is best security practice to always completely rebuild any compromised machine from a trusted backup or software distribution CDs. Unless you have real experts in house with time to burn you cannot be sure that a machine does not harbour dormant malicious code. The only time I would make an exception to this is with a documented virus/worm where there is a published clean up procedure or program.
To prevent a re-occurence you must harden your servers and keep up to date with security fixes. There are many guides around to do this. SANS have one of the better guides though does cost $49.
http://www.sansstore.org/ (look for the step by step security linux guide).
Other detailed information at:
http://www.linuxsecurity.com/
> I'd give a 1000 points if it's possible
300's enough. If you are led to separate questions, you'll have the points to use for them
300's enough. If you are led to separate questions, you'll have the points to use for them
I don't have much time toady, maybe later, to read this. Get system upgraded as best you can according to distro, keep in touch with linux topic area here at EE, and pay visit to CERT site which has been more attentive to linux hacks lately.
If you identified intruder, firewall the IP, even an IP range if needed to get some stability for one. You might need to keep one, or a copy, offline for troubleshooting. Don't neglect the common tools like tcpdump to track the live parts for what is being done, such as malformed something or other. SNMP is late trick, don't let that in from internet. Use local only.
If you identified intruder, firewall the IP, even an IP range if needed to get some stability for one. You might need to keep one, or a copy, offline for troubleshooting. Don't neglect the common tools like tcpdump to track the live parts for what is being done, such as malformed something or other. SNMP is late trick, don't let that in from internet. Use local only.
Sample extract:
-
CERT Advisory CA-2002-06 Vulnerabilities in Various Implementations of the RADIUS Protocol
Original release date: March 4, 2002
...
Various RADIUS servers and clients permit the passing of
vendor-specific and user-specific attributes. Several
implementations of RADIUS fail to check the vendor-length of
vendor-specific attributes. It is possible to cause a denial of
service against RADIUS servers with a malformed vendor-specific
attribute.
RADIUS servers and clients fail to validate the vendor-length
inside vendor-specific attributes. The vendor-length shouldn't be
less than 2. If vendor-length is less than 2, the RADIUS server (or
client) calculates the attribute length as a negative number. The
attribute length is then used in various functions. In most RADIUS
servers the function that performs this calculation is rad_recv()
or radrecv(). Some applications may use the same logic to validate
user-specific attributes and be vulnerable via the same method.
-
CERT Advisory CA-2002-06 Vulnerabilities in Various Implementations of the RADIUS Protocol
Original release date: March 4, 2002
...
Various RADIUS servers and clients permit the passing of
vendor-specific and user-specific attributes. Several
implementations of RADIUS fail to check the vendor-length of
vendor-specific attributes. It is possible to cause a denial of
service against RADIUS servers with a malformed vendor-specific
attribute.
RADIUS servers and clients fail to validate the vendor-length
inside vendor-specific attributes. The vendor-length shouldn't be
less than 2. If vendor-length is less than 2, the RADIUS server (or
client) calculates the attribute length as a negative number. The
attribute length is then used in various functions. In most RADIUS
servers the function that performs this calculation is rad_recv()
or radrecv(). Some applications may use the same logic to validate
user-specific attributes and be vulnerable via the same method.
Another:
-
1. Multiple Vulnerabilities in SNMP
Numerous vulnerabilities have been reported in multiple vendors'
SNMP implementations. These vulnerabilities may allow unauthorized
privileged access, denial-of-service attacks, or cause unstable
behavior. If your site uses SNMP in any capacity, the CERT/CC
encourages you to read this advisory and follow the advice
provided in the Solution section. In addition to this advisory, we
also have an FAQ on SNMP vulnerabilities.
CERT Advisory CA-2002-03:
Multiple Vulnerabilities In Many Implementations of
the Simple Network Management Protocol (SNMP)
http://www.cert.org/advisories/CA-2002-03.html
Simple Network Management Protocol (SNMP) Vulnerabilities
Frequently Asked Questions (FAQ)
http://www.cert.org/tech_tips/snmp_faq.html
-
1. Multiple Vulnerabilities in SNMP
Numerous vulnerabilities have been reported in multiple vendors'
SNMP implementations. These vulnerabilities may allow unauthorized
privileged access, denial-of-service attacks, or cause unstable
behavior. If your site uses SNMP in any capacity, the CERT/CC
encourages you to read this advisory and follow the advice
provided in the Solution section. In addition to this advisory, we
also have an FAQ on SNMP vulnerabilities.
CERT Advisory CA-2002-03:
Multiple Vulnerabilities In Many Implementations of
the Simple Network Management Protocol (SNMP)
http://www.cert.org/advisories/CA-2002-03.html
Simple Network Management Protocol (SNMP) Vulnerabilities
Frequently Asked Questions (FAQ)
http://www.cert.org/tech_tips/snmp_faq.html
Another newbie: NetFilter flaw (check build)
Quoting (with some links to thank IDG (re)source):
-----------------
http://www.nwfusion.com/news/2002/0301linuxhole.html
Software bug threatens select Linux systems
By Joris Evers
IDG News Service, 03/01/02
A flaw in the Netfilter firewall component of various versions of the Linux kernel could put systems running the open-source operating system at risk, experts warned this week.
A bug in part of the Netfilter system could result in unwanted ports for inbound traffic being opened on the firewall, effectively opening the door for hackers, according to a warning issued by the Netfilter team earlier this week. The flawed part of the system is meant to monitor chat requests sent and received on the Internet Relay Chat (IRC) network.
Advertisement:
Netfilter is part of all Linux kernels from Versions 2.4.14 to 2.1.18-pre8, according to the Netfilter statement. Red Hat, one of the largest Linux vendors, said in an alert that Red Hat Linux Versions 7.1 and 7.2 are vulnerable, but noted that the flawed IRC connection-tracking component of Netfilter is not used in default installations.
The IDG News Service is a Network World affiliate.
Related links
Netfilter's patch for the flaw is available here,
http://www.netfilter.org/files/2002-02-25-irc-dcc-mask.patch
while Red Hat's patches are available here.
http://www.redhat.com/support/errata/RHSA-2002-028.html
Forum: The SNMP hole
What are you doing about the newly discovered hole in this key protocol?
Network World Security and Bug Patch Alert
Sign up for our free e-mail newsletter.
Quoting (with some links to thank IDG (re)source):
-----------------
http://www.nwfusion.com/news/2002/0301linuxhole.html
Software bug threatens select Linux systems
By Joris Evers
IDG News Service, 03/01/02
A flaw in the Netfilter firewall component of various versions of the Linux kernel could put systems running the open-source operating system at risk, experts warned this week.
A bug in part of the Netfilter system could result in unwanted ports for inbound traffic being opened on the firewall, effectively opening the door for hackers, according to a warning issued by the Netfilter team earlier this week. The flawed part of the system is meant to monitor chat requests sent and received on the Internet Relay Chat (IRC) network.
Advertisement:
Netfilter is part of all Linux kernels from Versions 2.4.14 to 2.1.18-pre8, according to the Netfilter statement. Red Hat, one of the largest Linux vendors, said in an alert that Red Hat Linux Versions 7.1 and 7.2 are vulnerable, but noted that the flawed IRC connection-tracking component of Netfilter is not used in default installations.
The IDG News Service is a Network World affiliate.
Related links
Netfilter's patch for the flaw is available here,
http://www.netfilter.org/files/2002-02-25-irc-dcc-mask.patch
while Red Hat's patches are available here.
http://www.redhat.com/support/errata/RHSA-2002-028.html
Forum: The SNMP hole
What are you doing about the newly discovered hole in this key protocol?
Network World Security and Bug Patch Alert
Sign up for our free e-mail newsletter.
SANS on SNMP (port info):
=========================
SANS FLASH ALERT: Widespread SNMP Vulnerability
2:30 PM EST 12 February, 2002
Note: This is preliminary data! If you have additional information, please send it to us at snmp@sans.org
In a few minutes wire services and other news sources will begin breaking a story about widespread vulnerabilities in SNMP (Simple Network Management Protocol). Exploits of the vulnerability cause systems to fail or to be taken over. The vulnerability can be found in more than a hundred manufacturers' systems and is very widespread - millions of routers and other systems are involved.
Your leadership is needed in making sure that all systems for which you have any responsibility are protected. To do that, first ensure that SNMP is turned off. If you absolutely must run SNMP, get the patch from your hardware or software vendor. They are all working on patches right now. It also makes sense for you to filter traffic destined for SNMP ports (assuming the system doing the filtering is patched).
To block SNMP access, block traffic to ports 161 and 162 for tcp and udp. In addition, if you are using Cisco, block udp for port 1993.
The problems were caused by programming errors that have been in the SNMP implementations for a long time, but only recently discovered.
CERT/CC is taking the lead on the process of getting the vendors to get their patches out. Additional information is posted at
http://www.cert.org/advisories/CA-2002-03.html
Two final notes.
Note 1: Turning off SNMP was one of the strong recommendations in the Top 20 Internet Security Vulnerabilities that the FBI's NIPC and SANS and the Federal CIO Council issued on October 1, 2001. If you didn't take that action then, now might be a good time to correct the rest of the top 20 as well as the SNMP problem. The Top 20 document is posted at http://www.sans.org/top20.htm
Note 2: If you have Cisco routers (that's true for 85% of our readers) you are going to have to patch them to fix this problem. This is a great time to make the other fixes that will protect your Cisco routers from an increasingly common set of increasingly bad attacks.
A great new free tool will be announced on Thursday that checks Cisco routers, finds most problems, and provides specific guidance on fixing each problem it finds. We've scheduled a web broadcast for Thursday afternoon at 1 PM EST (18:00 UTC) to tell you about it and how to get it.
=========================
SANS FLASH ALERT: Widespread SNMP Vulnerability
2:30 PM EST 12 February, 2002
Note: This is preliminary data! If you have additional information, please send it to us at snmp@sans.org
In a few minutes wire services and other news sources will begin breaking a story about widespread vulnerabilities in SNMP (Simple Network Management Protocol). Exploits of the vulnerability cause systems to fail or to be taken over. The vulnerability can be found in more than a hundred manufacturers' systems and is very widespread - millions of routers and other systems are involved.
Your leadership is needed in making sure that all systems for which you have any responsibility are protected. To do that, first ensure that SNMP is turned off. If you absolutely must run SNMP, get the patch from your hardware or software vendor. They are all working on patches right now. It also makes sense for you to filter traffic destined for SNMP ports (assuming the system doing the filtering is patched).
To block SNMP access, block traffic to ports 161 and 162 for tcp and udp. In addition, if you are using Cisco, block udp for port 1993.
The problems were caused by programming errors that have been in the SNMP implementations for a long time, but only recently discovered.
CERT/CC is taking the lead on the process of getting the vendors to get their patches out. Additional information is posted at
http://www.cert.org/advisories/CA-2002-03.html
Two final notes.
Note 1: Turning off SNMP was one of the strong recommendations in the Top 20 Internet Security Vulnerabilities that the FBI's NIPC and SANS and the Federal CIO Council issued on October 1, 2001. If you didn't take that action then, now might be a good time to correct the rest of the top 20 as well as the SNMP problem. The Top 20 document is posted at http://www.sans.org/top20.htm
Note 2: If you have Cisco routers (that's true for 85% of our readers) you are going to have to patch them to fix this problem. This is a great time to make the other fixes that will protect your Cisco routers from an increasingly common set of increasingly bad attacks.
A great new free tool will be announced on Thursday that checks Cisco routers, finds most problems, and provides specific guidance on fixing each problem it finds. We've scheduled a web broadcast for Thursday afternoon at 1 PM EST (18:00 UTC) to tell you about it and how to get it.
Once learning of keeping up with patches, you might want to consider review of your favored distro, for keeping up with current needs. Side example, here a Cert list of people keeping up on one flaw:
http://www.cert.org/advisories/CA-2002-03.html#vendors/
http://www.cert.org/advisories/CA-2002-03.html#vendors/
looks like
18732 ? R 241:29 ./s3 0 200.215.129.6 1 56650 1
is the trojan backdoor that was left behind to access the servers
if the hacker uses a rootkit it's possible that you won't see everything (replaced ps, netstat, ...)
is 200.215.129.6 one of your servers ?
check if there is a process listening on port 56650
(netstat -lutnp)
my guess is ./s3
also the /usr/local/sbin/sshd looks strange
mine is located in /usr/sbin not /usr/local/sbin
but that could depend on the Linux version...
the problem with a hacked system is that it's almost impossible to find all the tools/bakdoors/replaced cmds and repair the system
best thing to do is to completely reinstall the servers from scratch (backup data ofcourse)
before you reconnect the new installed systems, make
shure you patched with ALL the latest updates, otherwise
they will get in again using the same vulnerabilities !!
Filip
18732 ? R 241:29 ./s3 0 200.215.129.6 1 56650 1
is the trojan backdoor that was left behind to access the servers
if the hacker uses a rootkit it's possible that you won't see everything (replaced ps, netstat, ...)
is 200.215.129.6 one of your servers ?
check if there is a process listening on port 56650
(netstat -lutnp)
my guess is ./s3
also the /usr/local/sbin/sshd looks strange
mine is located in /usr/sbin not /usr/local/sbin
but that could depend on the Linux version...
the problem with a hacked system is that it's almost impossible to find all the tools/bakdoors/replaced cmds and repair the system
best thing to do is to completely reinstall the servers from scratch (backup data ofcourse)
before you reconnect the new installed systems, make
shure you patched with ALL the latest updates, otherwise
they will get in again using the same vulnerabilities !!
Filip
use different box, different addy, to build server, then when firmed up, replace
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
I might point out that I've never had a hacked server that was secured using the above practices (although I am by no means claiming it would be impossible to hack such a server). I do *not* make this claim on servers secured *after* they were hacked. There is always a nagging doubt in the back of my mind about allowing previously hacked boxes back into production - what if there is some library modified to capture my passwords, etc (I'm not paranoid here - I've seen it happen). In my opinoin, if you do not reinstall, you are begging for trouble.
If you have a spare box, FlamingSword's method is probably the least painful.
-Jon
If you have a spare box, FlamingSword's method is probably the least painful.
-Jon
ASKER
Great recomendations I have installed from fresh media and implemented several of the items mentioned. Thanks for all the comments live and learn!
ASKER