Solved

Can't stop hacker from gaining root access to my web server! please help

Posted on 2011-09-13
12
732 Views
Last Modified: 2012-05-12
yesterday I went to ssh into my web server like a normal day when it said my root password was invalid. I logged in with another account with no problem but still couldn't get root access. I checked the logs and saw a new account was created with root privileges and my root password was changed both from an IP based in Asia. I'm the only one with access to my dedicated server and I'm in the US.

I called my hosting company and had them reset the server and root password. I logged in deleted that new account, limited ssh access to only my one limited user account, changed passwords for that account and a different one for root, changed the port ssh runs on, setup iptables to block all connections except on port 80 (http), 443 (https) and the new port I set for ssh. I thought to myself that now it's completely secure. except today I login and find all my logs are deleted, a new account is created again with root access and my iptables rules have been cleared. I did everything again but no doubt the hacker will be back in soon.

what should I do? I'm at a loss. I'm in progress of copying all my valuable files off my server to do a fresh install and start over. my websites average over 150K pageviews per day so I don't want to go offline if possible. How can I find how he's getting in? please help. not sure what to do here.

I run Cent OS linux and apache.
0
Comment
Question by:itcdr
  • 2
  • 2
  • 2
  • +6
12 Comments
 
LVL 7

Expert Comment

by:unSpawn
ID: 36528139
I went to ssh into my web server like a normal day when it said my root password was invalid.
It is an error to allow root to log in over any (hostile) network: always harden SSH, always log in using an unprivileged account and pubkey auth and limit brute forcing using 'fail2ban' or equivalent.


I don't want to go offline if possible.
Common these days (years actually) is for attackers to leverage vulnerabilities in the web stack (database, web server, but more than that: any badly configured or stale software running on top of it) to gain execute commands and gain access. It was an error was to believe that you could "fix" things without having to find out how the intruder gained root access. Unacceptable or not but now your first priority is not serving content but mitigation, regaining control. Please read CERT Intruder Detection Checklist to stand at least a small chance of finding out more details and inform other users they should consider their login/passwords for the system compromised as well.

If you suspect live processes you can glean information from then I suggest you first gather information running something like
( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /path/to/data.txt

Open in new window

(adjust path and file name) and then firewall off any 'net-facing publicly accessible services that are not vital to machine management: database, web server, FTP, etc, etc (basically you only want SSH access) else shut them down so they are no longer publicly accessible. (As you run Centos you should also verify system integrity using
bin/rpm -Vva 2>&1|/bin/grep -v "\.\{8\}" 2>&1> /path/to/rpmvfy.log

Open in new window

.) Attach /path/to/data.txt if you think it shows anything interesting.

* Note log deletion does not always mean files aren't written to unless a service was restarted. The 'lsof' output ('grep dele|/path/to/data.txt') may may still list file descriptors you could copy out. If you made regular off-site backups of your system and daemon logs then you should run everything together through Logwatch on a known secure, separate machine. They might provide clues.

If no logs remain (and note I'm trying to avoid mentioning forensics as that would require more time and effort than it is usually worth) then your only option is to export verifiable data (no binaries!) to a separate system for backup and verification purposes, install a brand new system with up to date software, harden the machine and its services and only then import your data after having verified each item. After that test your setup for potential problems before you go live again.
0
 
LVL 13

Expert Comment

by:khairil
ID: 36528147
Hi,

Changing ssh port do a little protection, these guys are smart, they will scan all you ports and use any vulnerabilities in your OS, Web Server and applications.

It is a big NO, NO of having root access remotely. You should not use root unless you are at the console itself, turn off remote access to root, http://centos.org/docs/4/4.5/Security_Guide/s2-wstation-privileges-noroot.html and http://www.howtogeek.com/howto/linux/security-tip-disable-root-ssh-login-on-linux and also allow only specific IP address to access SSH server, http://www.cyberciti.biz/tips/linux-iptables-4-block-all-incoming-traffic-but-allow-ssh.html
also use sudo instead if you need root access.

 Of course other tips is not to simple password and make it long. Also remember to patch your server.

Apache also can also be vulnerable. You need to patch it with all security patches and also enable mod_security on it, http://www.cyberciti.biz/faq/rhel-fedora-centos-httpd-mod_security-configuration.

You need to properly config you htaccess for proper authentication and authorization access, http://httpd.apache.org/docs/1.3/howto/auth.html

Follow guides on security best practice of you web applicaiton. I assume that you are using PHP so here is way to do it, http://phpsec.org/projects/guide
0
 
LVL 6

Accepted Solution

by:
JRoyse earned 250 total points
ID: 36529501
khairil said it, here it is another way: http://www.debian-administration.org/articles/87

Keep SSH more secure by only allowing your source network (like your home/work & another place)
# /etc/hosts.allow
sshd: 1.2.3.0/255.255.255.0
sshd: 192.168.0.0/255.255.255.0

Open in new window

# /etc/hosts.deny
sshd: ALL

Open in new window


Does your
~/.ssh/authorized_keys

Open in new window

have anything in it - for the root account?  

It is very likely there is a rootkit/trojan process installed to phone home and re-initialize a connection like a back-door.  

You could check for files created/modified in the last 15 days:
cd /
find . -ctime -15 -print #created
find . -mtime -15 -print #modified

Open in new window


0
 
LVL 1

Author Comment

by:itcdr
ID: 36530953
thanks guys for the responses. so far the hacker doesn't appear to have gotten back in today since I deleted his account the second time and setup a more strict firewall yesterday but I'm not convinced yet he can't get back in. and need to figure out what was done

trying all the suggestions and I'll post as I find anything. keep em coming if you think of anything more I can do.

@JRoyse, good idea with the commands searching for newly created/modified files. I'll go through all my files before pulling my backup off the server to make sure nothing is in there from the hacker. no ~/.ssh/authorized_keys file exists.

take a look at this though:

# cat .bash_history

dir
cd msf3
./msfconsole
./msfupdate
./msfconsole
yum remove ruby
./msfconsole
yum install ruby
./msfconsole
gem
rubygem
yum install rubygems
yum install ruby*
gem
yum install rubyg*
yum install gem*
ruby
clera
clear
nmap nguyentandung.org
http://nhipsongso.tuoitre.vn/Nhip-song-so/455316/Tran-lan-web-mao-danh-lanh-dao.html
111/tcp open  rpcbind
cd ../
dir
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz
http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz
tar xzvf rubygems-1.8.10.tgz
cd rubygems-1.8.10
ruby setup.rb
yum remove ruby
clear
cd ../msf3
./msfconsole
cd ../
dir
wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p290.tar.gz
yum install -y rpm-build rpmdevtools
rpmdev-setuptree
yum install -y rpmdev-setuptree
cd ~/rpmbuild/SOURCES
dir
curl https://raw.github.com/imeyer/ruby-1.9.2-rpm/master/ruby19.spec > ruby19.spec
rpmbuild -bb ruby19.spec
cd /usr/src/redhat/SOURCES/
wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p180.tar.gz
cd ../SPECs
cd ../SPECS
curl https://raw.github.com/imeyer/ruby-1.9.2-rpm/master/ruby19.spec > ruby19.spec
rpmbuild -bb ruby19.spec
cd ../SOURCES
dir
wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p290.tar.gz
cd ../SPECS
rpmbuild -bb ruby19.spec
yum install tcl-devel
rpmbuild -bb ruby19.spec
cd ../RPMS
rpm -Uvh ruby-1.9.2p180-2.ruby-1.9.2p180-2.i386.rpm
dir
cd x86*
dir
rpm -Uvh ruby-1.9.2p180-2.ruby-1.9.2p180-2.i386.rpm
rpm -Uvh ruby-1.9.2p290-2.x86_64.rpm
cd /pentest/msf3
dir
./msfupdate
cd ../
dir
cd rubygems*
ruby setup.rb
ruby -e 'require "rubygems"; require_gem "postgres";'
cd ../msf3
./msfconsole
exit


looks like these are the commands the hacker initiated while logged-in as root on my server. can you help me figure out what this all means?
0
 
LVL 12

Expert Comment

by:hfraser
ID: 36531151
msfconsole is part of the Metasploit toolkit. It's written in ruby, hence the downloading and building of the ruby packages.

Metaploit's a tool for exploiting vulnerabilities. It may be that the intruder's using your site to launch attacks on other sites.
0
 
LVL 5

Expert Comment

by:RizyDeWino
ID: 36532551
Just to add on what others have already said, run rootkit hunter and see if it reports any binaries have been modified.


In general at this stage, you can spend lot of time identifying what all is compromised on OS level or if hacker left any back door.

After any root level compromise like you have , its Always highly recommended to resetup fresh system and then migrate the user/application data to it after proper scanning/auditing to make sure nothing already compromised or with back doors is being copied over.

For fresh install, follow the guidelines experts have given above , like disabling direct root, changing default ssh port 22 to some other uncommon port like 3872 etc . Similar take other steps for secure setup like latest stable OS , Apache and other applications. Additional you can use firewall like CSF to have tight control and intrusion detection/protection. Listing all will be too long, but if you search on 'Securing SSH'  or Securing new Linux Install' etc on google, you will find many well written articles that describe the complete set of security steps you can take to make the server secure and lot less vulnerable to attacks/hacks.
0
Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

 
LVL 5

Assisted Solution

by:hvillanu
hvillanu earned 250 total points
ID: 36533498
Hi,
As already suggest you, disable remote access to root account.
change (or not) ssh port
But in a scan port could suffer a brute force attack, so you can configure a couple iptables rules to discover this brute force attack and block these IPs.


#!/bin/sh
## IPTABLES SCRIPT

# <Flush and blah... your rules>

# $INTIF is the LAN NIC and $EXTIF is the Internet NIC
INTIF="ethX"
EXTIF="ethY"
# Create a policy to "mark" the attacks
iptables -N BLACKLIST
iptables -F BLACKLIST
iptables -A BLACKLIST -j LOG --log-prefix "LOG_BLACK: "
iptables -A BLACKLIST -j DROP

#Blacklist attempt to access SSH
#Setup rate limits
# rate 1 - 2 attemps in 20 secs
iptables -A INPUT -i $EXTIF -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -i $EXTIF -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 20 --hitcount 2 --rttl --name SSH -j BLACKLIST
# rate 2 - 2 attemps in 60 seconds
iptables -A INPUT -i $EXTIF -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 --rttl --name SSH -j BLACKLIST
# rate 3 - 14 attemps in 200 secs
iptables -A INPUT -i $EXTIF -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 200 --hitcount 14 --rttl --name SSH -j BLACKLIST

Now Depending on your logs locations/files you can search specific access or drops that you previously mark

cat /some/path/log/file | grep 'DPT=22' > /Docs/Firewall/analisis/exit-file_ssh
cat /some/path/log/file | grep DROP >  /Docs/Firewall/analisis/exit-file_drop
cat /some/path/log/file | grep BLACK >  /Docs/Firewall/analisis/exit-file_black
cat /some/path/log/file | grep sshd > /Docs/Firewall/analisis/exit-file_sshd
cat /some/path/log/file | grep 'failure for' >  /Docs/Firewall/analisis/exit-file_failure
cat /some/path/log/file | grep 'Invalid user' >  /Docs/Firewall/analisis/exit-file_invalid
cat /some/path/log/file | grep Accepted >  /Docs/Firewall/analisis/exit-file_accepted

--hope helps--
0
 
LVL 3

Expert Comment

by:maxchow
ID: 36534600
Guy,

I think you have to review the services on your server, use the minimal authority rule, use netstat to find out unnecessary services and close them down first, default installation of nowaday linux distro given too much rubbish

2nd, disable root access to ssh, this can be done by changing /etc/ssh/sshd_config, change PermitRootLogin to no.

you can also use the chkrootkit to check whether the hacker has injected any rootkit.

3rd give root a damn crazy password and keep it safe. and use sudo for most of your other jobs, don't create a operator account like operator, support, technical, etc, this is crazy.
0
 
LVL 1

Author Comment

by:itcdr
ID: 36546521
update. I did a fresh install of centos and followed many suggestions here. but the hacker got in again. luckily though this time I was able to figure out how. one of my smaller sites I recently purchased has a nulled/pirated version of bulletin for its forum. When searching those files I found the hacker found a security hole in the forum and was uploading viruses. the forum has since been disabled but not sure now the hacker has done.

I may re-install linux again. although I may have to check all my code and database to make sure there isn't a security hole there now because of the hacker
0
 
LVL 6

Expert Comment

by:JRoyse
ID: 36548782
If the software got exploited the hacker should only be able to get the permission of the webservice/php/database.  You would have to also have a local-privilege exploit vulnerability on your operating system to allow someone under the "apache" or "httpd" user to bump up to Root access.  If you are curious SElinux configuration on CentOS may help with that or telling yum to keep the OS updated also.
0
 
LVL 3

Expert Comment

by:maxchow
ID: 36554339
Agree with JRoyse comment
0
 
LVL 15

Expert Comment

by:It breaks therefore I am
ID: 36554836
Just a thought, do you connect to your Linux box from a PC running Windows using something like Putty ? If so maybe you should check your PC for viruses. Many times I have seen sites exploited because a developer has given away FTP passwords in this way and if you have root access that's going to be a lot worse.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now