Link to home
Start Free TrialLog in
Avatar of Chris Kenward
Chris KenwardFlag for United Kingdom of Great Britain and Northern Ireland

asked on

tmp directory security breach in CentOS 5.5

Even though I've mounted /tmp in my fstab file like this:

/tmp                /tmp           bind           nosuid,noexec,bind   0 0

and rebooted my server, I'm still finding that someone is coming in, uploading a VOIP server to my system, and running it from the /tmp folder. I keep on deleting the directory structure that's holding the VOIP hack (called ALOHA?) but almost every morning it's back and it doesn't appear I can do much about it. The /tmp folder looks like this now:

 User generated image
I didn't put the .htaccess file in the /tmp folder so I'm a little suspicious of it. It contains the following:

 User generated image
I'm fairly new to all things CentOS/Linux so I have no idea whether the rest of the stuff in there looks "right" or not. I also have no idea how to find out how these files are being uploaded and by whom or which script on the server is allowing this type of thing to happen.

Any help would be sincerely appreciated. This is a production server with some 30 websites on it that we host for our customers, so I have to be extra cautious with any potential fixes which might stop the sites from working.

Looking forward to any help you might give.

Avatar of Chris Kenward
Chris Kenward
Flag of United Kingdom of Great Britain and Northern Ireland image


As an update, here is a copy of my /etc/fstab file, in case I haven't done the "harden /tmp folder" the right way.

Chris User generated image
It seems your server has been hacked. Since you have no idea of how deep the hack is (rootkit) the general rule is to reinstall the machine from scratch. Before putting the machine in production update all services to the latest version, disable unnecessary services and implement good firewall rules.

Several system files (executables) are probably changed on your machine and hackers probably have remote root access to the machine so you can not do much with the current machine.


There is a file name security.log in /var/log/ check the logs in that file, you may find the commands which the hacker has been executed.

Just 2 years back my server was also hacked and I have check the security.log and found all the changes which hacker made and I have revert them.

So same would be helpful for you. Otherwise do the same thing which Blaz have suggested.

@Blaz: Really loathe to reinstall the server unless I absolutely have to. Is there any way to discover how many files have been changed on the server? You mentioned rootkit - is this what I should use?

@upanwar: Have checked the secure.log and related (older) files and there is nothing in there about files changing, only about logins, whether successful or not. Is there a way to make this log file more verbose?

Rootkit is the type of attack which (can) change all the system files in such a way that you can't see the changes. For example it replaces the ps command (list of all running processes) so that it does not display the processes that are running for backdoor entrance or that are currently being executed by the hacker. Also ls command with which you found offending files on /tmp/ could be changed so that you would not see the hack relevant files at all.

Because it is quite difficult to know how deeply your system has been changed it is quite difficult to repair it.

In CentOS you have one way of telling what was changed - "rpm -Va" command displays changes to the files that were installed with rpm packages (most programs in CentOS). You should be concentrating on files in /usr/bin, /usr/sbin, /bin, /sbin folders that have MD5 checksum different (the 3rd character is "5" instead of "."). However depending on the smartness of the attacker he could change this program not to report any changes or changed the database so that the test will pass...
Hi Blaz

This is what the command comes up with:

S.?.....    /usr/lib/
S.5....T  c /etc/printcap
SM5....T  c /etc/sysconfig/iptables-config
S.5....T  c /etc/logrotate.conf
S.5....T  c /etc/httpd/conf/httpd.conf
SM5....T    /usr/lib/xorg/modules/drivers/
SM5....T    /usr/lib/xorg/modules/input/
S.5....T  c /etc/yum/pluginconf.d/priorities.conf
..5....T  c /etc/inittab
S.5....T  c /etc/sysconfig/network-scripts/ifcfg-lo
S.5....T  c /etc/sysctl.conf
SM5....T    /usr/libexec/webmin/status/
SM5....T    /usr/libexec/webmin/status/
S.5....T  c /etc/vsftpd/vsftpd.conf
S.5....T  c /etc/sysconfig/vncservers
S.5....T  c /etc/ssh/sshd_config
S.5....T  c /etc/php.ini
S.5....T  c /etc/httpd/conf.d/ssl.conf
S.5....T  c /etc/ppp/chap-secrets
S.5....T  c /etc/ppp/pap-secrets
S.5....T  c /etc/smartd.conf
S.?.....    /usr/lib/
S.5....T  c /etc/xml/catalog
S.5....T  c /usr/share/sgml/docbook/xmlcatalog
S.5....T  c /etc/updatedb.conf
S.5....T  c /etc/sysconfig/system-config-securitylevel

Is there anything in there that looks onerous?

Avatar of farzanj
As experts already suggested, you need to re-install the server.

What happens is that when your security is compromised, your commands change too.  For example, your ps is corrupted and wouldn't show you a hacker process.  Your ls would not show you hacker's files.

In furture, to make sure that your system is not hacked you can follow a few recommendations.

Strictly use RPMS.  Use RPM verification scripts to periodically to check if your RPMs are changed.  Use RPM certificates to test their integrity.

Secure services and ports.  If possible, use SELinux.  There is a steep learning curve.

Use file system ACLs.

Use multiple levels of security of services, iptables, TCP wrappers, Xinet, SELinux, etc.  NEVER leave a port open if you don't need it.  Block it using iptables.

Do not install services you don't need.  If they exist, they should not be running and their ports should be block.  If possible, these policies should be protected using SELinux.

Do not use insecure services like FTP and telnet where passwords travel unencrypted.  Use strong encryptions.
farzanj, How do I tell if those commands have indeed been compromised? They aren't in the list above are they? So checksums have not changed? Sorry - I'm new to all this and this doesn't really make much sense to me I'm afraid

Avatar of farzanj
Flag of Canada image

Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Make sure no folder on your system is world writable.  Make permissions very restrictive.  You can use SELinux to make policies so that no one can make any file/folder world readable/writable
Hi there, farzanj

Thanks very much for the time you're taking with this - much appreciated. I have no idea how to do any of the things you're talking about above so it would be great if you could elaborate on them for me?

Firstly, re the checksums. The list of files I put up above are the only files which appear to have the "5" instead of the "." at position 3, so I'm assuming for the moment (or hoping! <g>) that the LS and other commands have not been compromised.

I've now stopped the SSHD service completely on this server, in the hope it'll stop whoever is coming in from uploading the files again.

I run VSFTPD for the FTP server but I have set all users to be chrooted in their own folders, or at least I'm pretty sure I've done that correctly.

How would I start by redirecting all access to root other than through SUDO? I've never heard of SUDO so in words of one syllable if you would? ;) That would be a good start I guess.

As an aside, I've installed rkhunter and rootcheck and run them both. There are a few warnings but mostly in the scripts area. I've read somewhere that the scripts are a false alarm but once again I'm doing all this from an experience standpoint. I'll go off and do some research!

Thanks again
You are using vsftp server.  It is insecure and password and all communication travels over the network unencrypted.  So any packet sniffing software (tcpdump, ethereal, snort, wireshark, etc.) may be used to see user ID's and passwords.  So, you have given hackers a spot on your computer.  Now you are testing their expertise how they would get out of chroot jail.

SSHD is secure and ideally should have been used.  Well, you know your priorities.  So I cannot judge that.  But most security guys would agree that SSH should be used instead of VSFTPD

Sudo is a utility  to execute privileged commands by an ordinary users.  To configure it do, visudo as root.  The file has sample configurations.  You can allow any user to use sudo with or without password.

Suppose user1 wants to be root user.  He would issue the following command
sudo su -

This would enable him to become a root user.  So root user will have no password, and ftp or sshd would be configured to restrict any root logins.  So any unknown user, say, chris, would login and after logging in would become root, eliminating the need to use root password.
Some RPM verification Background:


Hi farzanj

OK - I have implemented the SUDO option on the server and it appears to be working OK. I can reboot the server and login using my personal account but cannot login as root. If I want to execute any root commands I have to type SUDO first, then my personal account password, then the command. Thanks for that.

Next my question revolves around VSFTPD. I realise that it is not secure but how do I get all those web designers whose sites are being hosted on this server upload their files without using an FTP package? Can you upload using SSH?

At this point, because I'm scared that whoever has been coming in was using SSH, I've stopped the service on that server for now. I'm the only one who needs it (other than for the FTP uploading of files) so I guess it's not missed right now.

If you could elaborate on how folks would use SSH to upload their files, that would be super.

Thanks again - I'm feeling much less worried about this server now that it's starting to look as though it's going to he "harder" than it was before.

Hi Chris,

For sudo you don't need to reboot the server.  For most Linux operations, there should be no need for reboot.  Try login from your user and if you are happy and satisfied, you should remove the root password from /etc/shadow file.  Instead of password hash, put "!!"

If you are inviting files from unknown users, you have to use ftp so that everyone uses anonymous ftp and uploads files.  If you know the users, who they are and you created a custom account for each one, then why not have them upload files through ssh or https or whatever secure service.

See, telnet and ftp is a pair, getting console and transferring files.  Likewise ssh and scp/sftp makes a pair.  Similarly, you can still put them into chroot jail.  Here is one link to start looking
When you successfully implement/test it, tell your users to start uploading files using sftp/scp.

One more thing.  Use simple iptables rules to block the ports of your server that you don't use.  You will have to be careful though.  But unmonitored open ports may be exploted.

Thanks very much for the ongoing help with this one. I'll research the wrappers and close as many ports as possible in an effort to harden further.

I don't know how to use SELinux but will begin research on that and who knows....?

You are most welcome!  Very few people know SELinux but it is extremely beneficial.  Makes system extremely hard to break.
I think I spoke too soon about being concerned... I've just been back on the server and the /tmp/aloha directory is back! I'm gutted - I thought the SUDO thing would have stopped that, but obviously this person is getting in via some other method and transferring the files back to the server anytime he wants to.

Is ftp still running?
Did you block other unimportant ports?
There may be a script running on your service that is doing it to drive you nuts.
Is there any cron job that is doing that?

cat /var/spool/cron/*

cat /var/spool/cron*/*/*
Whichever works.

Don't get frustrated.  Security needs a lot of patience.
You can certainly open a new more specific question after your research
Hi farzanj

Should I open another question for this ongoing problem? Seems a little unfair to continue here now that this one's closed?

I've just stopped the vsftpd service and a look through the running processes doesn't look as though there are any other ftp services running at this time.

I'll do the port examination thing now and see what I can block or close.

I'll ask a related question and begin with this post. Please would you look out for it?

Thanks for your help.
It would be better to open a question for each area where you need help.  That way you would find more many responses.
Done! Thanks, farzanj