Was I just hacked?

Posted on 2003-03-31
Medium Priority
Last Modified: 2008-02-01
I'm not a very expirenced Linux user.  I have a box co-located with my ISP.  My website was running fine not 1/2 hour ago and my friend just phoned to say that my site was down.

Apache wouldn't start up because another program was already bound to port 80.  I took a look at the programs that were running and noticed the following programs being run as root that I didn't initate and were started at the same time my site when down.

I notice they are copying their own setpasswd, md5sum, and run programs into my system as well as setting the permissions of some things as well.  Here is the partial ps listing:

root      6739  0.0  0.1  1376  264 ?        T    16:01   0:00 cp -f setpasswd /usr/bin/setpasswd
root      6742  0.0  0.1  1376  264 ?        T    16:01   0:00 cp -f tools/sum /usr/bin/md5sum
root      6747  0.0  0.0     0    0 ?        Z    16:01   0:00 [cp <defunct>]
root      6753  0.0  0.0     0    0 ?        Z    16:01   0:00 [cp <defunct>]
root      6806  0.0  0.1  1376  264 ?        T    16:02   0:00 cp run /usr/bin/run
root      6816  0.0  0.0     0    0 ?        Z    16:02   0:00 [cp <defunct>]
root      6822  0.0  0.1  1480  320 ?        T    16:02   0:00 /usr/sbin/chattr -AacdisSu /sbin/init
root      6827  0.0  0.0     0    0 ?        Z    16:02   0:00 [chattr <defunct>]
root      6828  0.0  0.1  1480  320 ?        T    16:02   0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk
root      6833  0.0  0.0     0    0 ?        Z    16:02   0:00 [chattr <defunct>]
root      6867  0.0  0.1  1360  268 ?        T    16:03   0:00 mkdir -p /etc/ssh
root      6872  0.0  0.0     0    0 ?        Z    16:03   0:00 [mkdir <defunct>]
root      6879  0.0  0.1  1848  344 ?        T    16:03   0:00 /usr/X11R6/.zk/xfs -p 122
root      6881  0.0  0.0     0    0 ?        Z    16:03   0:00 [xfs <defunct>]
root      7050  0.0  0.1  1480  320 ?        T    16:04   0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg
root      7054  0.0  0.0     0    0 ?        Z    16:04   0:00 [chattr <defunct>]
root      7258  0.0  0.1  1404  284 ?        S    16:10   0:00 ls --color=tty

I tried doing a reboot of my system and now it will not come back up.... Waiting on my ISP to get someone to reset the server manually.

Has anyone seen this type of alteration before?  Thanks
Question by:vancetech
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions

Author Comment

ID: 8242673
My system hangs just after initailizing my hostname.  If I hit ctrl-c it tries to continue and displays this information:

INIT: Entering Runlevel: 3
/etc/rc.d/rc :  /var/run/runlevel.dir : Read Only filesystem.

This occurs with any other programs that try to touch a lock file in /var/lock/subsys or /var/lock/run.
LVL 19

Accepted Solution

Gabriel Orozco earned 750 total points
ID: 8245973
well... yes. you where hacked.

what to do from here?
1.- Hope you have a backup. if you do not, then backup all YOUR DATA not your programs, which can be compromised, as the hacker was replacing and hidding programs.
2.- reinstall. do this with a new, updated version of your operating system. do not left programs which you are not confident of.
3.- apply a very good firewall and check what services you are running. those programs running services accesable from outside should be maintained up to date always.

mmhh... good luck with this =/
LVL 14

Expert Comment

ID: 8246228
And apply all security patches for your OS and other software.
And harden your box with something like Bastille Linux (www.bastille-linux.org) and by following the Center for Internet Security recommendations (www.cisecurity.org).
WatchGuard's M Series Appliances - Miecom Approved

WatchGuard's newest M series appliances were put to the test by Miercom.  We had great results and outperformed all of our competitors in both stateless and stateful traffic throghput scenarios! Ready to see how your UTM appliance stacked up? Download the Miercom Report!


Expert Comment

ID: 8253679
I saw some lines with chattr and got interested.

See link for ref.

Somewhat according to the little log you provided,

16:02   0:00 /usr/sbin/chattr -AacdisSu /sbin/init

Suggests the hacker took away all rights on /sbin/init
Would explain why u cant reboot.

16:02   0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk

Added I flag to init and init.zk

16:03   0:00 mkdir -p /etc/ssh

Make directory /etc/ssh,  suggests s/he want to
use ssh in.

16:03   0:00 /usr/X11R6/.zk/xfs -p 122

Suggests s/he is using Xwindows on port 122 as an backdoor.

16:04   0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg

So, take away the line in inetcfg, which mention port122,
otherwise s/he will still have access to yr server.
There should somewhere else on your server-logs some
entries of an "buffer overflow", or some other hints
to which s/he got root access in the first place.

do as the others told you,  reinstall os,  make sure you apply the latest patches for it.  Ask your co-host for help.  You might need physical access to your server.

When you find the logs for which s/he got access first, send all info to CERT or follow the procedures your co-host have for such events.

Best of times!

Expert Comment

ID: 8261226
I agree you should rebuild your machine from the ground up.  Read up on what services to leave running on your machine - you didn't mention what distro you're running. Keep in mind that you only want to have network services running that are necessary for the box to operate.  So, for your webserver - probably just apache and ssh. If you use something like mysql or other network listeners you're going to have to make sure you firewall them off to the outside world.

You can see what services are listening with the lsof tool (netstat has probably been hacked):

#> lsof -i -n

make sure you stop services like nfslock and portmap and install the newest version of the distribution you can get your hands on (i.e. download a new version).

I would highly recommend using shiny new copies of apache, ssh (whatever pkg your running), ftpd (or just start using sftp if you're not already)

and one more very important thing - keep your machine off the wire while you're doing the install and initial services config - i.e

1 - disconnect network
2 - reinstall OS
3 - turn off _all_ listening network services
4 - reconnect network
5 - patch patch patch!
6 - reconfigure bare min of listeners like apache and ssh

good luck!

<happyfunjuice #>

Author Comment

ID: 8263958
You guys all rock... Thanks for the informative help!  I realized quickly that a complete re-install was in order.

Im on a Redhat 7.2 linux box, and run a full gamit of services for my clients.  Http, ftp, pop, smtp, mysql etc, and I have learned that keeping those up to date is imperative.

I'm finally back online, fully updated with the 7.2 OS and all the security updates for RPMs.

Thank you again, you've all been very helpful!

Author Comment

ID: 8263968
Thanks for prompt reply!

Featured Post

More Than Just A Video Library

Train for your certification. Learn the latest DevOps tools. Grow your skillset to do better work.

At Linux Academy, we release new training modules every week so you'll always be up to date on the latest tech.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Hello EE, Today we will learn how to send all your network traffic through Tor which is useful to get around censorship and being tracked all together to a certain degree. This article assumes you will be using Linux, have a minimal knowledge of …
BIND is the most widely used Name Server. A Name Server is the one that translates a site name to it's IP address. There is a new bug in BIND (https://kb.isc.org/article/AA-01272), affecting all versions of BIND 9 from BIND 9.1.0 (inclusive) thro…
Monitoring a network: how to monitor network services and why? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the philosophy behind service monitoring and why a handshake validation is critical in network monitoring. Software utilized …
If you’ve ever visited a web page and noticed a cool font that you really liked the look of, but couldn’t figure out which font it was so that you could use it for your own work, then this video is for you! In this Micro Tutorial, you'll learn yo…
Suggested Courses

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question