Was I just hacked?

I'm not a very expirenced Linux user.  I have a box co-located with my ISP.  My website was running fine not 1/2 hour ago and my friend just phoned to say that my site was down.

Apache wouldn't start up because another program was already bound to port 80.  I took a look at the programs that were running and noticed the following programs being run as root that I didn't initate and were started at the same time my site when down.

I notice they are copying their own setpasswd, md5sum, and run programs into my system as well as setting the permissions of some things as well.  Here is the partial ps listing:

root      6739  0.0  0.1  1376  264 ?        T    16:01   0:00 cp -f setpasswd /usr/bin/setpasswd
root      6742  0.0  0.1  1376  264 ?        T    16:01   0:00 cp -f tools/sum /usr/bin/md5sum
root      6747  0.0  0.0     0    0 ?        Z    16:01   0:00 [cp <defunct>]
root      6753  0.0  0.0     0    0 ?        Z    16:01   0:00 [cp <defunct>]
root      6806  0.0  0.1  1376  264 ?        T    16:02   0:00 cp run /usr/bin/run
root      6816  0.0  0.0     0    0 ?        Z    16:02   0:00 [cp <defunct>]
root      6822  0.0  0.1  1480  320 ?        T    16:02   0:00 /usr/sbin/chattr -AacdisSu /sbin/init
root      6827  0.0  0.0     0    0 ?        Z    16:02   0:00 [chattr <defunct>]
root      6828  0.0  0.1  1480  320 ?        T    16:02   0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk
root      6833  0.0  0.0     0    0 ?        Z    16:02   0:00 [chattr <defunct>]
root      6867  0.0  0.1  1360  268 ?        T    16:03   0:00 mkdir -p /etc/ssh
root      6872  0.0  0.0     0    0 ?        Z    16:03   0:00 [mkdir <defunct>]
root      6879  0.0  0.1  1848  344 ?        T    16:03   0:00 /usr/X11R6/.zk/xfs -p 122
root      6881  0.0  0.0     0    0 ?        Z    16:03   0:00 [xfs <defunct>]
root      7050  0.0  0.1  1480  320 ?        T    16:04   0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg
root      7054  0.0  0.0     0    0 ?        Z    16:04   0:00 [chattr <defunct>]
root      7258  0.0  0.1  1404  284 ?        S    16:10   0:00 ls --color=tty

I tried doing a reboot of my system and now it will not come back up.... Waiting on my ISP to get someone to reset the server manually.

Has anyone seen this type of alteration before?  Thanks
Who is Participating?
Gabriel OrozcoConnect With a Mentor Solution ArchitectCommented:
well... yes. you where hacked.

what to do from here?
1.- Hope you have a backup. if you do not, then backup all YOUR DATA not your programs, which can be compromised, as the hacker was replacing and hidding programs.
2.- reinstall. do this with a new, updated version of your operating system. do not left programs which you are not confident of.
3.- apply a very good firewall and check what services you are running. those programs running services accesable from outside should be maintained up to date always.

mmhh... good luck with this =/
vancetechAuthor Commented:
My system hangs just after initailizing my hostname.  If I hit ctrl-c it tries to continue and displays this information:

INIT: Entering Runlevel: 3
/etc/rc.d/rc :  /var/run/runlevel.dir : Read Only filesystem.

This occurs with any other programs that try to touch a lock file in /var/lock/subsys or /var/lock/run.
And apply all security patches for your OS and other software.
And harden your box with something like Bastille Linux (www.bastille-linux.org) and by following the Center for Internet Security recommendations (www.cisecurity.org).
Will You Be GDPR Compliant by 5/28/2018?

GDPR? That's a regulation for the European Union. But, if you collect data from customers or employees within the EU, then you need to know about GDPR and make sure your organization is compliant by May 2018. Check out our preparation checklist to make sure you're on track today!

I saw some lines with chattr and got interested.

See link for ref.

Somewhat according to the little log you provided,

16:02   0:00 /usr/sbin/chattr -AacdisSu /sbin/init

Suggests the hacker took away all rights on /sbin/init
Would explain why u cant reboot.

16:02   0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk

Added I flag to init and init.zk

16:03   0:00 mkdir -p /etc/ssh

Make directory /etc/ssh,  suggests s/he want to
use ssh in.

16:03   0:00 /usr/X11R6/.zk/xfs -p 122

Suggests s/he is using Xwindows on port 122 as an backdoor.

16:04   0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg

So, take away the line in inetcfg, which mention port122,
otherwise s/he will still have access to yr server.
There should somewhere else on your server-logs some
entries of an "buffer overflow", or some other hints
to which s/he got root access in the first place.

do as the others told you,  reinstall os,  make sure you apply the latest patches for it.  Ask your co-host for help.  You might need physical access to your server.

When you find the logs for which s/he got access first, send all info to CERT or follow the procedures your co-host have for such events.

Best of times!
I agree you should rebuild your machine from the ground up.  Read up on what services to leave running on your machine - you didn't mention what distro you're running. Keep in mind that you only want to have network services running that are necessary for the box to operate.  So, for your webserver - probably just apache and ssh. If you use something like mysql or other network listeners you're going to have to make sure you firewall them off to the outside world.

You can see what services are listening with the lsof tool (netstat has probably been hacked):

#> lsof -i -n

make sure you stop services like nfslock and portmap and install the newest version of the distribution you can get your hands on (i.e. download a new version).

I would highly recommend using shiny new copies of apache, ssh (whatever pkg your running), ftpd (or just start using sftp if you're not already)

and one more very important thing - keep your machine off the wire while you're doing the install and initial services config - i.e

1 - disconnect network
2 - reinstall OS
3 - turn off _all_ listening network services
4 - reconnect network
5 - patch patch patch!
6 - reconfigure bare min of listeners like apache and ssh

good luck!

<happyfunjuice #>
vancetechAuthor Commented:
You guys all rock... Thanks for the informative help!  I realized quickly that a complete re-install was in order.

Im on a Redhat 7.2 linux box, and run a full gamit of services for my clients.  Http, ftp, pop, smtp, mysql etc, and I have learned that keeping those up to date is imperative.

I'm finally back online, fully updated with the 7.2 OS and all the security updates for RPMs.

Thank you again, you've all been very helpful!
vancetechAuthor Commented:
Thanks for prompt reply!
All Courses

From novice to tech pro — start learning today.