• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 289
  • Last Modified:

Linux Box Intrusion

Also posted to OS Linux but more points and more info posted here.

VA Linux 6.2.4, Internet Server runing Sendmail, DNS, radius, and HTTP/FTP services.  Two identical
servers have been hacked by the same people.

ps aux >ps shows this command running

"./s3 0 1 56650 1 800"

The machine that has crashed was also showing these processes running.

1035 ?        D      8:40 (nfsiod)
1036 ?        R      7:57 (nfsiod)
2585 ?        D      8:25 (nfsiod)
13011 ?        Z      0:00 [osdbo <defunct>]
9964 ?        D      6:58 (nfsiod)
19379 ?        S      0:00 httpd
13296 ?        D      7:39 (nfsiod)
9611 ?        D      6:57 (nfsiod)
1245 ?        D      6:58 (nfsiod)
5954 ?        Z      0:00 [osdbo <defunct>]
1339 ?        D      6:01 (nfsiod)
2475 ?        D      6:04 (nfsiod)
31666 ?        D      5:48 (nfsiod)
6106 ?        D      5:36 (nfsiod)
23795 ?        Z      0:00 [osdbo <defunct>]
28778 ?        D      1:19 (nfsiod)
30060 ?        D      1:05 (nfsiod)
27633 ?        Z      0:00 [osdbo <defunct>]
10136 ?        D      0:59 (nfsiod)
29318 ?        D      0:49 (nfsiod)
18732 ?        R    241:29 ./s3 0 1 56650 1
16670 ?        D      0:16 (nfsiod)
31264 ?        D      0:13 (nfsiod)
12316 ?        S      0:00 (nfsiod)
7377 ?        S      0:00 /usr/local/sbin/sshd
7420 ttyp0    S      0:00 -bash
29402 ?        R      0:00 (nfsiod)
29403 ?        R      0:00 (nfsiod)
29404 ?        R      0:00 (nfsiod)
29405 ?        R      0:00 (nfsiod)
29406 ?        R      0:00 (nfsiod)
29407 ?        R      0:00 (nfsiod)
29408 ?        R      0:00 (nfsiod)
29409 ?        R      0:00 (nfsiod)
29410 ?        R      0:00 (nfsiod)
29411 ?        R      0:00 (nfsiod)
29412 ?        R      0:00 (nfsiod)
29413 ?        R      0:00 (nfsiod)
29414 ?        R      0:00 sh -c
29420 ?        R      0:00 (nfsiod)
29421 ?        R      0:00 (nfsiod)
29422 ?        R      0:00 (nfsiod)
29423 ?        R      0:00 (nfsiod)

The intruder apparently created the following folders:

/dev/. /s3  (that is a space after the period)
/dev/. /OSDBO
/dev/. /OSDBO2

The second machine where we have observed the

"./s3 0 1 56650 1 800" command

has not had the nfsiod processes showing.

There have been various IP addresses associated with this effort all so far from the .br domain.

The s3 folder appears to have the contents of the /dev directory. We rebooted one machine, and it's toast, would not boot back up.  When the intruder attaches, we can disconnect him and he'll reattach almost immediately.  

Does anybody recognize this hack process?  

Can you me tell what it's doing, and most importantly how
to reverse the effects before this system crashes and prevent it from happening again?

  • 7
  • 2
  • 2
  • +3
1 Solution
fallonsupportAuthor Commented:
I'd give a 1000 points if it's possible. System seems to only allow 300 when posting.

Somebody else may recognise the hack.  In terms of what you need to do.

It is best security practice to always completely rebuild any compromised machine from a trusted backup or software distribution CDs.  Unless you have real experts in house with time to burn you cannot be sure that a machine does not harbour dormant malicious code.  The only time I would make an exception to this is with a documented virus/worm where there is a published clean up procedure or program.

To prevent a re-occurence you must harden your servers and keep up to date with security fixes.  There are many guides around to do this.  SANS have one of the better guides though does cost $49.

http://www.sansstore.org/  (look for the step by step security linux guide).

Other detailed information at:

> I'd give a 1000 points if it's possible

300's enough. If you are led to separate questions, you'll have the points to use for them
Managed Security Services Webinar - March 15

Selecting the right managed security services platform to grow your business can be a huge undertaking. Join WatchGuard and Frost & Sullivan in an upcoming webinar as we dive into the key elements of selecting a vendor platform and partnership to fuel a successful MSSP business.

I don't have much time toady, maybe later, to read this. Get system upgraded as best you can according to distro, keep in touch with linux topic area here at EE, and pay visit to CERT site which has been more attentive to linux hacks lately.

If you identified intruder, firewall the IP, even an IP range if needed to get some stability for one. You might need to keep one, or a copy, offline for troubleshooting. Don't neglect the common tools like tcpdump to track the live parts for what is being done, such as malformed  something or other. SNMP is late trick, don't let that in from internet. Use local only.
Sample extract:
CERT Advisory CA-2002-06 Vulnerabilities in Various Implementations of the RADIUS Protocol
Original release date: March 4, 2002
 Various   RADIUS   servers   and  clients  permit  the  passing  of
     vendor-specific     and     user-specific    attributes.    Several
     implementations  of  RADIUS  fail  to  check  the  vendor-length of
     vendor-specific  attributes.  It  is  possible to cause a denial of
     service  against  RADIUS  servers  with a malformed vendor-specific

     RADIUS  servers  and  clients  fail  to  validate the vendor-length
     inside  vendor-specific  attributes. The vendor-length shouldn't be
     less than 2. If vendor-length is less than 2, the RADIUS server (or
     client)  calculates  the attribute length as a negative number. The
     attribute  length is then used in various functions. In most RADIUS
     servers  the  function that performs this calculation is rad_recv()
     or  radrecv(). Some applications may use the same logic to validate
     user-specific attributes and be vulnerable via the same method.
 1. Multiple Vulnerabilities in SNMP

       Numerous  vulnerabilities  have been reported in multiple vendors'
       SNMP implementations. These vulnerabilities may allow unauthorized
       privileged  access,  denial-of-service  attacks, or cause unstable
       behavior.  If  your  site  uses  SNMP in any capacity, the CERT/CC
       encourages  you  to  read  this  advisory  and  follow  the advice
       provided in the Solution section. In addition to this advisory, we
       also have an FAQ on SNMP vulnerabilities.

                CERT Advisory CA-2002-03:
          Multiple Vulnerabilities In Many Implementations of
          the Simple Network Management Protocol (SNMP)

                Simple Network Management Protocol (SNMP) Vulnerabilities
                Frequently Asked Questions (FAQ)
Another newbie: NetFilter flaw (check build)
Quoting (with some links to thank IDG (re)source):

Software bug threatens select Linux systems
By Joris Evers
IDG News Service, 03/01/02

A flaw in the Netfilter firewall component of various versions of the Linux kernel could put systems running the open-source operating system at risk, experts warned this week.

A bug in part of the Netfilter system could result in unwanted ports for inbound traffic being opened on the firewall, effectively opening the door for hackers, according to a warning issued by the Netfilter team earlier this week. The flawed part of the system is meant to monitor chat requests sent and received on the Internet Relay Chat (IRC) network.

Netfilter is part of all Linux kernels from Versions 2.4.14 to 2.1.18-pre8, according to the Netfilter statement. Red Hat, one of the largest Linux vendors, said in an alert that Red Hat Linux Versions 7.1 and 7.2 are vulnerable, but noted that the flawed IRC connection-tracking component of Netfilter is not used in default installations.

The IDG News Service is a Network World affiliate.

Related links

Netfilter's patch for the flaw is available here,
while Red Hat's patches are available here.

Forum: The SNMP hole
What are you doing about the newly discovered hole in this key protocol?

Network World Security and Bug Patch Alert
Sign up for our free e-mail newsletter.
SANS on SNMP (port info):

SANS FLASH ALERT: Widespread SNMP Vulnerability
2:30 PM EST 12 February, 2002

Note: This is preliminary data! If you have additional information, please send it to us at snmp@sans.org

In a few minutes wire services and other news sources will begin breaking a story about widespread vulnerabilities in SNMP (Simple Network Management Protocol).  Exploits of the vulnerability cause systems to fail or to be taken over.  The vulnerability can be found in more than a hundred manufacturers' systems and is very widespread - millions of routers and other systems are involved.

Your leadership is needed in making sure that all systems for which you have any responsibility are protected. To do that, first ensure that SNMP is turned off. If you absolutely must run SNMP, get the patch from your hardware or software vendor. They are all working on patches right now. It also makes sense for you to filter traffic destined for SNMP ports (assuming the system doing the filtering is patched).

To block SNMP access, block traffic to ports 161 and 162 for tcp and udp.  In addition, if you are using Cisco, block udp for port 1993.

The problems were caused by programming errors that have been in the SNMP implementations for a long time, but only recently discovered.

CERT/CC is taking the lead on the process of getting the vendors to get their patches out.  Additional information is posted at

Two final notes.

Note 1:  Turning off SNMP was one of the strong recommendations in the Top 20 Internet Security Vulnerabilities that the FBI's NIPC and SANS and the Federal CIO Council issued on October 1, 2001.  If you didn't take that action then, now might be a good time to correct the rest of the top 20 as well as the SNMP problem.  The Top 20 document is posted at http://www.sans.org/top20.htm

Note 2:  If you have Cisco routers (that's true for 85% of our readers) you are going to have to patch them to fix this problem. This is a great time to make the other fixes that will protect your Cisco routers from an increasingly common set of increasingly bad attacks.

A great new free tool will be announced on Thursday that checks Cisco routers, finds most problems, and provides specific guidance on fixing each problem it finds.  We've scheduled a web broadcast for Thursday afternoon at 1 PM EST (18:00 UTC) to tell you about it and how to get it.
Once learning of keeping up with patches, you might want to consider review of your favored distro, for keeping up with current needs.  Side example, here a Cert list of people keeping up on one flaw:
looks like

18732 ?        R    241:29 ./s3 0 1 56650 1

is the trojan backdoor that was left behind to access the servers

if the hacker uses a rootkit it's possible that you won't see everything (replaced ps, netstat, ...)

is one of your servers ?

check if there is a process listening on port 56650
(netstat -lutnp)
my guess is ./s3

also the /usr/local/sbin/sshd looks strange
mine is located in /usr/sbin not /usr/local/sbin
but that could depend on the Linux version...

the problem with a hacked system is that it's almost impossible to find all the tools/bakdoors/replaced cmds and repair the system

best thing to do is to completely reinstall the servers from scratch (backup data ofcourse)

before you reconnect the new installed systems, make
shure you patched with ALL the latest updates, otherwise
they will get in again using the same vulnerabilities !!


use different box, different addy, to build server, then when firmed up, replace
Looks like an nfs hack to me - once they found the vulnerability on one box, they used it on the other (hence the lack of dead nfsiod processes).

Whoever said that you should reinstall is right.  There is no end to malicious crap that can reside hidden on your system unless you run tripwire and regularly back up the tripwire database (sounds like you didn't/don't).

If the above is not an option (and you may be conviced later that it *is* an option, after your box gets hacked over and over), proceed thusly...

Disconnect the machine from the network, so you can't get re-hacked while you are fixing things.

Re-install all system binaries with the latest versions, so you can be sure ps, netstat, ls, top, etc are working properly.

Portscan every port (1-65535), on the system, both udp and tcp, from an adjacent host (or even better, from a fake network you set up to mimic the attacker's IP).  Find and eliminate all unnecessary (and possibly hidden) servers.  If y ou re-installed all the system binaries, they can't hide for very long (usually).

Change all public services to run on non-standard ports, if possible (obviously, this will be a problem for public web/ftp/email services).

Don't run any services with known exploits on public interfaces (cough*cough*NFS*cough)

Install an agressive IDS to actively block attacks in progress (i.e. we run our SSH on a non-standard port, and our IDS blocks anyone trying to cennect on port 22)

Run tripwire to monitor unauthorized changes on your filesystem.

Block unauthorized ports at the edge/firewall (except for those designed to specifically trip your IDS).

Plug machine back into network.


I might point out that I've never had a hacked server that was secured using the above practices (although I am by no means claiming it would be impossible to hack such a server).  I do *not* make this claim on servers secured *after* they were hacked.  There is always a nagging doubt in the back of my mind about allowing previously hacked boxes back into production - what if there is some library modified to capture my passwords, etc (I'm not paranoid here - I've seen it happen).  In my opinoin, if you do not reinstall, you are begging for trouble.

If you have a spare box, FlamingSword's method is probably the least painful.


fallonsupportAuthor Commented:
Great recomendations I have installed from fresh media and implemented several of the items mentioned. Thanks for all the comments live and learn!
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Get 10% Off Your First Squarespace Website

Ready to showcase your work, publish content or promote your business online? With Squarespace’s award-winning templates and 24/7 customer service, getting started is simple. Head to Squarespace.com and use offer code ‘EXPERTS’ to get 10% off your first purchase.

  • 7
  • 2
  • 2
  • +3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now