vancetech
asked on
Was I just hacked?
I'm not a very expirenced Linux user. I have a box co-located with my ISP. My website was running fine not 1/2 hour ago and my friend just phoned to say that my site was down.
Apache wouldn't start up because another program was already bound to port 80. I took a look at the programs that were running and noticed the following programs being run as root that I didn't initate and were started at the same time my site when down.
I notice they are copying their own setpasswd, md5sum, and run programs into my system as well as setting the permissions of some things as well. Here is the partial ps listing:
root 6739 0.0 0.1 1376 264 ? T 16:01 0:00 cp -f setpasswd /usr/bin/setpasswd
root 6742 0.0 0.1 1376 264 ? T 16:01 0:00 cp -f tools/sum /usr/bin/md5sum
root 6747 0.0 0.0 0 0 ? Z 16:01 0:00 [cp <defunct>]
root 6753 0.0 0.0 0 0 ? Z 16:01 0:00 [cp <defunct>]
root 6806 0.0 0.1 1376 264 ? T 16:02 0:00 cp run /usr/bin/run
root 6816 0.0 0.0 0 0 ? Z 16:02 0:00 [cp <defunct>]
root 6822 0.0 0.1 1480 320 ? T 16:02 0:00 /usr/sbin/chattr -AacdisSu /sbin/init
root 6827 0.0 0.0 0 0 ? Z 16:02 0:00 [chattr <defunct>]
root 6828 0.0 0.1 1480 320 ? T 16:02 0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk
root 6833 0.0 0.0 0 0 ? Z 16:02 0:00 [chattr <defunct>]
root 6867 0.0 0.1 1360 268 ? T 16:03 0:00 mkdir -p /etc/ssh
root 6872 0.0 0.0 0 0 ? Z 16:03 0:00 [mkdir <defunct>]
root 6879 0.0 0.1 1848 344 ? T 16:03 0:00 /usr/X11R6/.zk/xfs -p 122
root 6881 0.0 0.0 0 0 ? Z 16:03 0:00 [xfs <defunct>]
root 7050 0.0 0.1 1480 320 ? T 16:04 0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg
root 7054 0.0 0.0 0 0 ? Z 16:04 0:00 [chattr <defunct>]
root 7258 0.0 0.1 1404 284 ? S 16:10 0:00 ls --color=tty
I tried doing a reboot of my system and now it will not come back up.... Waiting on my ISP to get someone to reset the server manually.
Has anyone seen this type of alteration before? Thanks
Apache wouldn't start up because another program was already bound to port 80. I took a look at the programs that were running and noticed the following programs being run as root that I didn't initate and were started at the same time my site when down.
I notice they are copying their own setpasswd, md5sum, and run programs into my system as well as setting the permissions of some things as well. Here is the partial ps listing:
root 6739 0.0 0.1 1376 264 ? T 16:01 0:00 cp -f setpasswd /usr/bin/setpasswd
root 6742 0.0 0.1 1376 264 ? T 16:01 0:00 cp -f tools/sum /usr/bin/md5sum
root 6747 0.0 0.0 0 0 ? Z 16:01 0:00 [cp <defunct>]
root 6753 0.0 0.0 0 0 ? Z 16:01 0:00 [cp <defunct>]
root 6806 0.0 0.1 1376 264 ? T 16:02 0:00 cp run /usr/bin/run
root 6816 0.0 0.0 0 0 ? Z 16:02 0:00 [cp <defunct>]
root 6822 0.0 0.1 1480 320 ? T 16:02 0:00 /usr/sbin/chattr -AacdisSu /sbin/init
root 6827 0.0 0.0 0 0 ? Z 16:02 0:00 [chattr <defunct>]
root 6828 0.0 0.1 1480 320 ? T 16:02 0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk
root 6833 0.0 0.0 0 0 ? Z 16:02 0:00 [chattr <defunct>]
root 6867 0.0 0.1 1360 268 ? T 16:03 0:00 mkdir -p /etc/ssh
root 6872 0.0 0.0 0 0 ? Z 16:03 0:00 [mkdir <defunct>]
root 6879 0.0 0.1 1848 344 ? T 16:03 0:00 /usr/X11R6/.zk/xfs -p 122
root 6881 0.0 0.0 0 0 ? Z 16:03 0:00 [xfs <defunct>]
root 7050 0.0 0.1 1480 320 ? T 16:04 0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg
root 7054 0.0 0.0 0 0 ? Z 16:04 0:00 [chattr <defunct>]
root 7258 0.0 0.1 1404 284 ? S 16:10 0:00 ls --color=tty
I tried doing a reboot of my system and now it will not come back up.... Waiting on my ISP to get someone to reset the server manually.
Has anyone seen this type of alteration before? Thanks
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
And apply all security patches for your OS and other software.
And harden your box with something like Bastille Linux (www.bastille-linux.org) and by following the Center for Internet Security recommendations (www.cisecurity.org).
And harden your box with something like Bastille Linux (www.bastille-linux.org) and by following the Center for Internet Security recommendations (www.cisecurity.org).
I saw some lines with chattr and got interested.
See link for ref.
http://linuxcommand.org/man_pages/chattr1.html
Somewhat according to the little log you provided,
16:02 0:00 /usr/sbin/chattr -AacdisSu /sbin/init
Suggests the hacker took away all rights on /sbin/init
Would explain why u cant reboot.
16:02 0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk
Added I flag to init and init.zk
16:03 0:00 mkdir -p /etc/ssh
Make directory /etc/ssh, suggests s/he want to
use ssh in.
16:03 0:00 /usr/X11R6/.zk/xfs -p 122
Suggests s/he is using Xwindows on port 122 as an backdoor.
16:04 0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg
So, take away the line in inetcfg, which mention port122,
otherwise s/he will still have access to yr server.
There should somewhere else on your server-logs some
entries of an "buffer overflow", or some other hints
to which s/he got root access in the first place.
SUGGESTION:
do as the others told you, reinstall os, make sure you apply the latest patches for it. Ask your co-host for help. You might need physical access to your server.
When you find the logs for which s/he got access first, send all info to CERT or follow the procedures your co-host have for such events.
Best of times!
See link for ref.
http://linuxcommand.org/man_pages/chattr1.html
Somewhat according to the little log you provided,
16:02 0:00 /usr/sbin/chattr -AacdisSu /sbin/init
Suggests the hacker took away all rights on /sbin/init
Would explain why u cant reboot.
16:02 0:00 /usr/sbin/chattr +i /sbin/init /sbin/init.zk
Added I flag to init and init.zk
16:03 0:00 mkdir -p /etc/ssh
Make directory /etc/ssh, suggests s/he want to
use ssh in.
16:03 0:00 /usr/X11R6/.zk/xfs -p 122
Suggests s/he is using Xwindows on port 122 as an backdoor.
16:04 0:00 /usr/sbin/chattr +i /sbin/ifconfig /sbin/inetcfg
So, take away the line in inetcfg, which mention port122,
otherwise s/he will still have access to yr server.
There should somewhere else on your server-logs some
entries of an "buffer overflow", or some other hints
to which s/he got root access in the first place.
SUGGESTION:
do as the others told you, reinstall os, make sure you apply the latest patches for it. Ask your co-host for help. You might need physical access to your server.
When you find the logs for which s/he got access first, send all info to CERT or follow the procedures your co-host have for such events.
Best of times!
I agree you should rebuild your machine from the ground up. Read up on what services to leave running on your machine - you didn't mention what distro you're running. Keep in mind that you only want to have network services running that are necessary for the box to operate. So, for your webserver - probably just apache and ssh. If you use something like mysql or other network listeners you're going to have to make sure you firewall them off to the outside world.
You can see what services are listening with the lsof tool (netstat has probably been hacked):
#> lsof -i -n
make sure you stop services like nfslock and portmap and install the newest version of the distribution you can get your hands on (i.e. download a new version).
I would highly recommend using shiny new copies of apache, ssh (whatever pkg your running), ftpd (or just start using sftp if you're not already)
and one more very important thing - keep your machine off the wire while you're doing the install and initial services config - i.e
1 - disconnect network
2 - reinstall OS
3 - turn off _all_ listening network services
4 - reconnect network
5 - patch patch patch!
6 - reconfigure bare min of listeners like apache and ssh
good luck!
<happyfunjuice #>
You can see what services are listening with the lsof tool (netstat has probably been hacked):
#> lsof -i -n
make sure you stop services like nfslock and portmap and install the newest version of the distribution you can get your hands on (i.e. download a new version).
I would highly recommend using shiny new copies of apache, ssh (whatever pkg your running), ftpd (or just start using sftp if you're not already)
and one more very important thing - keep your machine off the wire while you're doing the install and initial services config - i.e
1 - disconnect network
2 - reinstall OS
3 - turn off _all_ listening network services
4 - reconnect network
5 - patch patch patch!
6 - reconfigure bare min of listeners like apache and ssh
good luck!
<happyfunjuice #>
ASKER
You guys all rock... Thanks for the informative help! I realized quickly that a complete re-install was in order.
Im on a Redhat 7.2 linux box, and run a full gamit of services for my clients. Http, ftp, pop, smtp, mysql etc, and I have learned that keeping those up to date is imperative.
I'm finally back online, fully updated with the 7.2 OS and all the security updates for RPMs.
Thank you again, you've all been very helpful!
Im on a Redhat 7.2 linux box, and run a full gamit of services for my clients. Http, ftp, pop, smtp, mysql etc, and I have learned that keeping those up to date is imperative.
I'm finally back online, fully updated with the 7.2 OS and all the security updates for RPMs.
Thank you again, you've all been very helpful!
ASKER
Thanks for prompt reply!
ASKER
INIT: Entering Runlevel: 3
/etc/rc.d/rc : /var/run/runlevel.dir : Read Only filesystem.
This occurs with any other programs that try to touch a lock file in /var/lock/subsys or /var/lock/run.