Link to home
Start Free TrialLog in
Avatar of itcdr
itcdrFlag for United States of America

asked on

Can't stop hacker from gaining root access to my web server! please help

yesterday I went to ssh into my web server like a normal day when it said my root password was invalid. I logged in with another account with no problem but still couldn't get root access. I checked the logs and saw a new account was created with root privileges and my root password was changed both from an IP based in Asia. I'm the only one with access to my dedicated server and I'm in the US.

I called my hosting company and had them reset the server and root password. I logged in deleted that new account, limited ssh access to only my one limited user account, changed passwords for that account and a different one for root, changed the port ssh runs on, setup iptables to block all connections except on port 80 (http), 443 (https) and the new port I set for ssh. I thought to myself that now it's completely secure. except today I login and find all my logs are deleted, a new account is created again with root access and my iptables rules have been cleared. I did everything again but no doubt the hacker will be back in soon.

what should I do? I'm at a loss. I'm in progress of copying all my valuable files off my server to do a fresh install and start over. my websites average over 150K pageviews per day so I don't want to go offline if possible. How can I find how he's getting in? please help. not sure what to do here.

I run Cent OS linux and apache.
Avatar of unSpawn
unSpawn
Flag of Sweden image

I went to ssh into my web server like a normal day when it said my root password was invalid.
It is an error to allow root to log in over any (hostile) network: always harden SSH, always log in using an unprivileged account and pubkey auth and limit brute forcing using 'fail2ban' or equivalent.


I don't want to go offline if possible.
Common these days (years actually) is for attackers to leverage vulnerabilities in the web stack (database, web server, but more than that: any badly configured or stale software running on top of it) to gain execute commands and gain access. It was an error was to believe that you could "fix" things without having to find out how the intruder gained root access. Unacceptable or not but now your first priority is not serving content but mitigation, regaining control. Please read CERT Intruder Detection Checklist to stand at least a small chance of finding out more details and inform other users they should consider their login/passwords for the system compromised as well.

If you suspect live processes you can glean information from then I suggest you first gather information running something like
( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /path/to/data.txt

Open in new window

(adjust path and file name) and then firewall off any 'net-facing publicly accessible services that are not vital to machine management: database, web server, FTP, etc, etc (basically you only want SSH access) else shut them down so they are no longer publicly accessible. (As you run Centos you should also verify system integrity using
bin/rpm -Vva 2>&1|/bin/grep -v "\.\{8\}" 2>&1> /path/to/rpmvfy.log

Open in new window

.) Attach /path/to/data.txt if you think it shows anything interesting.

* Note log deletion does not always mean files aren't written to unless a service was restarted. The 'lsof' output ('grep dele|/path/to/data.txt') may may still list file descriptors you could copy out. If you made regular off-site backups of your system and daemon logs then you should run everything together through Logwatch on a known secure, separate machine. They might provide clues.

If no logs remain (and note I'm trying to avoid mentioning forensics as that would require more time and effort than it is usually worth) then your only option is to export verifiable data (no binaries!) to a separate system for backup and verification purposes, install a brand new system with up to date software, harden the machine and its services and only then import your data after having verified each item. After that test your setup for potential problems before you go live again.
Hi,

Changing ssh port do a little protection, these guys are smart, they will scan all you ports and use any vulnerabilities in your OS, Web Server and applications.

It is a big NO, NO of having root access remotely. You should not use root unless you are at the console itself, turn off remote access to root, http://centos.org/docs/4/4.5/Security_Guide/s2-wstation-privileges-noroot.html and http://www.howtogeek.com/howto/linux/security-tip-disable-root-ssh-login-on-linux and also allow only specific IP address to access SSH server, http://www.cyberciti.biz/tips/linux-iptables-4-block-all-incoming-traffic-but-allow-ssh.html
also use sudo instead if you need root access.

 Of course other tips is not to simple password and make it long. Also remember to patch your server.

Apache also can also be vulnerable. You need to patch it with all security patches and also enable mod_security on it, http://www.cyberciti.biz/faq/rhel-fedora-centos-httpd-mod_security-configuration.

You need to properly config you htaccess for proper authentication and authorization access, http://httpd.apache.org/docs/1.3/howto/auth.html

Follow guides on security best practice of you web applicaiton. I assume that you are using PHP so here is way to do it, http://phpsec.org/projects/guide
ASKER CERTIFIED SOLUTION
Avatar of JRoyse
JRoyse
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of itcdr

ASKER

thanks guys for the responses. so far the hacker doesn't appear to have gotten back in today since I deleted his account the second time and setup a more strict firewall yesterday but I'm not convinced yet he can't get back in. and need to figure out what was done

trying all the suggestions and I'll post as I find anything. keep em coming if you think of anything more I can do.

@JRoyse, good idea with the commands searching for newly created/modified files. I'll go through all my files before pulling my backup off the server to make sure nothing is in there from the hacker. no ~/.ssh/authorized_keys file exists.

take a look at this though:

# cat .bash_history

dir
cd msf3
./msfconsole
./msfupdate
./msfconsole
yum remove ruby
./msfconsole
yum install ruby
./msfconsole
gem
rubygem
yum install rubygems
yum install ruby*
gem
yum install rubyg*
yum install gem*
ruby
clera
clear
nmap nguyentandung.org
http://nhipsongso.tuoitre.vn/Nhip-song-so/455316/Tran-lan-web-mao-danh-lanh-dao.html
111/tcp open  rpcbind
cd ../
dir
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz
http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz
tar xzvf rubygems-1.8.10.tgz
cd rubygems-1.8.10
ruby setup.rb
yum remove ruby
clear
cd ../msf3
./msfconsole
cd ../
dir
wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p290.tar.gz
yum install -y rpm-build rpmdevtools
rpmdev-setuptree
yum install -y rpmdev-setuptree
cd ~/rpmbuild/SOURCES
dir
curl https://raw.github.com/imeyer/ruby-1.9.2-rpm/master/ruby19.spec > ruby19.spec
rpmbuild -bb ruby19.spec
cd /usr/src/redhat/SOURCES/
wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p180.tar.gz
cd ../SPECs
cd ../SPECS
curl https://raw.github.com/imeyer/ruby-1.9.2-rpm/master/ruby19.spec > ruby19.spec
rpmbuild -bb ruby19.spec
cd ../SOURCES
dir
wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p290.tar.gz
cd ../SPECS
rpmbuild -bb ruby19.spec
yum install tcl-devel
rpmbuild -bb ruby19.spec
cd ../RPMS
rpm -Uvh ruby-1.9.2p180-2.ruby-1.9.2p180-2.i386.rpm
dir
cd x86*
dir
rpm -Uvh ruby-1.9.2p180-2.ruby-1.9.2p180-2.i386.rpm
rpm -Uvh ruby-1.9.2p290-2.x86_64.rpm
cd /pentest/msf3
dir
./msfupdate
cd ../
dir
cd rubygems*
ruby setup.rb
ruby -e 'require "rubygems"; require_gem "postgres";'
cd ../msf3
./msfconsole
exit


looks like these are the commands the hacker initiated while logged-in as root on my server. can you help me figure out what this all means?
msfconsole is part of the Metasploit toolkit. It's written in ruby, hence the downloading and building of the ruby packages.

Metaploit's a tool for exploiting vulnerabilities. It may be that the intruder's using your site to launch attacks on other sites.
Just to add on what others have already said, run rootkit hunter and see if it reports any binaries have been modified.


In general at this stage, you can spend lot of time identifying what all is compromised on OS level or if hacker left any back door.

After any root level compromise like you have , its Always highly recommended to resetup fresh system and then migrate the user/application data to it after proper scanning/auditing to make sure nothing already compromised or with back doors is being copied over.

For fresh install, follow the guidelines experts have given above , like disabling direct root, changing default ssh port 22 to some other uncommon port like 3872 etc . Similar take other steps for secure setup like latest stable OS , Apache and other applications. Additional you can use firewall like CSF to have tight control and intrusion detection/protection. Listing all will be too long, but if you search on 'Securing SSH'  or Securing new Linux Install' etc on google, you will find many well written articles that describe the complete set of security steps you can take to make the server secure and lot less vulnerable to attacks/hacks.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of maxchow
maxchow

Guy,

I think you have to review the services on your server, use the minimal authority rule, use netstat to find out unnecessary services and close them down first, default installation of nowaday linux distro given too much rubbish

2nd, disable root access to ssh, this can be done by changing /etc/ssh/sshd_config, change PermitRootLogin to no.

you can also use the chkrootkit to check whether the hacker has injected any rootkit.

3rd give root a damn crazy password and keep it safe. and use sudo for most of your other jobs, don't create a operator account like operator, support, technical, etc, this is crazy.
Avatar of itcdr

ASKER

update. I did a fresh install of centos and followed many suggestions here. but the hacker got in again. luckily though this time I was able to figure out how. one of my smaller sites I recently purchased has a nulled/pirated version of bulletin for its forum. When searching those files I found the hacker found a security hole in the forum and was uploading viruses. the forum has since been disabled but not sure now the hacker has done.

I may re-install linux again. although I may have to check all my code and database to make sure there isn't a security hole there now because of the hacker
If the software got exploited the hacker should only be able to get the permission of the webservice/php/database.  You would have to also have a local-privilege exploit vulnerability on your operating system to allow someone under the "apache" or "httpd" user to bump up to Root access.  If you are curious SElinux configuration on CentOS may help with that or telling yum to keep the OS updated also.
Agree with JRoyse comment
Just a thought, do you connect to your Linux box from a PC running Windows using something like Putty ? If so maybe you should check your PC for viruses. Many times I have seen sites exploited because a developer has given away FTP passwords in this way and if you have root access that's going to be a lot worse.