Solved

tmp directory security breach in CentOS 5.5

Posted on 2011-02-14
22
1,693 Views
Last Modified: 2012-05-11
Even though I've mounted /tmp in my fstab file like this:

/tmp                /tmp           bind           nosuid,noexec,bind   0 0

and rebooted my server, I'm still finding that someone is coming in, uploading a VOIP server to my system, and running it from the /tmp folder. I keep on deleting the directory structure that's holding the VOIP hack (called ALOHA?) but almost every morning it's back and it doesn't appear I can do much about it. The /tmp folder looks like this now:

 tmp folder after suspicious file removal
I didn't put the .htaccess file in the /tmp folder so I'm a little suspicious of it. It contains the following:

 Contents of .htaccess file
I'm fairly new to all things CentOS/Linux so I have no idea whether the rest of the stuff in there looks "right" or not. I also have no idea how to find out how these files are being uploaded and by whom or which script on the server is allowing this type of thing to happen.

Any help would be sincerely appreciated. This is a production server with some 30 websites on it that we host for our customers, so I have to be extra cautious with any potential fixes which might stop the sites from working.

Looking forward to any help you might give.

Chris
0
Comment
Question by:kenwardc
  • 10
  • 9
  • 2
  • +1
22 Comments
 

Author Comment

by:kenwardc
Comment Utility
As an update, here is a copy of my /etc/fstab file, in case I haven't done the "harden /tmp folder" the right way.

Cheers
Chris The /etc/fstab file
0
 
LVL 16

Expert Comment

by:Blaz
Comment Utility
It seems your server has been hacked. Since you have no idea of how deep the hack is (rootkit) the general rule is to reinstall the machine from scratch. Before putting the machine in production update all services to the latest version, disable unnecessary services and implement good firewall rules.

Several system files (executables) are probably changed on your machine and hackers probably have remote root access to the machine so you can not do much with the current machine.

0
 
LVL 12

Expert Comment

by:upanwar
Comment Utility
Hi,

There is a file name security.log in /var/log/ check the logs in that file, you may find the commands which the hacker has been executed.

Just 2 years back my server was also hacked and I have check the security.log and found all the changes which hacker made and I have revert them.

So same would be helpful for you. Otherwise do the same thing which Blaz have suggested.

0
 

Author Comment

by:kenwardc
Comment Utility
@Blaz: Really loathe to reinstall the server unless I absolutely have to. Is there any way to discover how many files have been changed on the server? You mentioned rootkit - is this what I should use?

@upanwar: Have checked the secure.log and related (older) files and there is nothing in there about files changing, only about logins, whether successful or not. Is there a way to make this log file more verbose?

Cheers
Chris
0
 
LVL 16

Expert Comment

by:Blaz
Comment Utility
Rootkit is the type of attack which (can) change all the system files in such a way that you can't see the changes. For example it replaces the ps command (list of all running processes) so that it does not display the processes that are running for backdoor entrance or that are currently being executed by the hacker. Also ls command with which you found offending files on /tmp/ could be changed so that you would not see the hack relevant files at all.

Because it is quite difficult to know how deeply your system has been changed it is quite difficult to repair it.

In CentOS you have one way of telling what was changed - "rpm -Va" command displays changes to the files that were installed with rpm packages (most programs in CentOS). You should be concentrating on files in /usr/bin, /usr/sbin, /bin, /sbin folders that have MD5 checksum different (the 3rd character is "5" instead of "."). However depending on the smartness of the attacker he could change this program not to report any changes or changed the database so that the test will pass...
0
 

Author Comment

by:kenwardc
Comment Utility
Hi Blaz

This is what the command comes up with:

S.?.....    /usr/lib/libpspell.so.15.1.3
S.5....T  c /etc/printcap
SM5....T  c /etc/sysconfig/iptables-config
S.5....T  c /etc/logrotate.conf
S.5....T  c /etc/httpd/conf/httpd.conf
SM5....T    /usr/lib/xorg/modules/drivers/vmware_drv.so
SM5....T    /usr/lib/xorg/modules/input/vmmouse_drv.so
S.5....T  c /etc/yum/pluginconf.d/priorities.conf
..5....T  c /etc/inittab
S.5....T  c /etc/sysconfig/network-scripts/ifcfg-lo
S.5....T  c /etc/sysctl.conf
SM5....T    /usr/libexec/webmin/status/module.info
SM5....T    /usr/libexec/webmin/status/ping-monitor.pl
S.5....T  c /etc/vsftpd/vsftpd.conf
S.5....T  c /etc/sysconfig/vncservers
S.5....T  c /etc/ssh/sshd_config
S.5....T  c /etc/php.ini
S.5....T  c /etc/httpd/conf.d/ssl.conf
S.5....T  c /etc/ppp/chap-secrets
S.5....T  c /etc/ppp/pap-secrets
S.5....T  c /etc/smartd.conf
S.?.....    /usr/lib/libnl.so.1.0-pre5
S.5....T  c /etc/xml/catalog
S.5....T  c /usr/share/sgml/docbook/xmlcatalog
S.5....T  c /etc/updatedb.conf
S.5....T  c /etc/sysconfig/system-config-securitylevel

Is there anything in there that looks onerous?

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
As experts already suggested, you need to re-install the server.

What happens is that when your security is compromised, your commands change too.  For example, your ps is corrupted and wouldn't show you a hacker process.  Your ls would not show you hacker's files.

In furture, to make sure that your system is not hacked you can follow a few recommendations.

Strictly use RPMS.  Use RPM verification scripts to periodically to check if your RPMs are changed.  Use RPM certificates to test their integrity.

Secure services and ports.  If possible, use SELinux.  There is a steep learning curve.

Use file system ACLs.

Use multiple levels of security of services, iptables, TCP wrappers, Xinet, SELinux, etc.  NEVER leave a port open if you don't need it.  Block it using iptables.

Do not install services you don't need.  If they exist, they should not be running and their ports should be block.  If possible, these policies should be protected using SELinux.

Do not use insecure services like FTP and telnet where passwords travel unencrypted.  Use strong encryptions.
0
 

Author Comment

by:kenwardc
Comment Utility
farzanj, How do I tell if those commands have indeed been compromised? They aren't in the list above are they? So checksums have not changed? Sorry - I'm new to all this and this doesn't really make much sense to me I'm afraid

Cheers
Chris
0
 
LVL 31

Accepted Solution

by:
farzanj earned 500 total points
Comment Utility
No Chris, your question makes absolute sense.

So are you saying that you kept track of the checksums and they are still matching for all the binaries and files?

I am not saying they are changed.  I am saying they could be.  Checksums are also good mechanism to check files are not changed but RPM verification in my opinion should always be used and is very strong.  It is also good because rpm database can be queried which would tell you what changed when, if at all.

My only point is that we should plan servers in a way that if the security is compromised, we would know.

Check your own ports and block any unnecessary open ports.  See if the strange behavior continues.  If it does, start bringing other services down (if you can do so) to see which one was causing problem.

Another way is to  use TCP Wrappers to log all the ssh sessions so that you would know who is using those.

Restrict all direct access to "root".  Make it accessible only through SUDO.  It keeps logs or you may have to configure it a little and this way you would know if someone is getting unnecessary root access.
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
Make sure no folder on your system is world writable.  Make permissions very restrictive.  You can use SELinux to make policies so that no one can make any file/folder world readable/writable
0
 

Author Comment

by:kenwardc
Comment Utility
Hi there, farzanj

Thanks very much for the time you're taking with this - much appreciated. I have no idea how to do any of the things you're talking about above so it would be great if you could elaborate on them for me?

Firstly, re the checksums. The list of files I put up above are the only files which appear to have the "5" instead of the "." at position 3, so I'm assuming for the moment (or hoping! <g>) that the LS and other commands have not been compromised.

I've now stopped the SSHD service completely on this server, in the hope it'll stop whoever is coming in from uploading the files again.

I run VSFTPD for the FTP server but I have set all users to be chrooted in their own folders, or at least I'm pretty sure I've done that correctly.

How would I start by redirecting all access to root other than through SUDO? I've never heard of SUDO so in words of one syllable if you would? ;) That would be a good start I guess.

As an aside, I've installed rkhunter and rootcheck and run them both. There are a few warnings but mostly in the scripts area. I've read somewhere that the scripts are a false alarm but once again I'm doing all this from an experience standpoint. I'll go off and do some research!

Thanks again
Chris
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 31

Expert Comment

by:farzanj
Comment Utility
You are using vsftp server.  It is insecure and password and all communication travels over the network unencrypted.  So any packet sniffing software (tcpdump, ethereal, snort, wireshark, etc.) may be used to see user ID's and passwords.  So, you have given hackers a spot on your computer.  Now you are testing their expertise how they would get out of chroot jail.

SSHD is secure and ideally should have been used.  Well, you know your priorities.  So I cannot judge that.  But most security guys would agree that SSH should be used instead of VSFTPD


Sudo is a utility  to execute privileged commands by an ordinary users.  To configure it do, visudo as root.  The file has sample configurations.  You can allow any user to use sudo with or without password.

Suppose user1 wants to be root user.  He would issue the following command
sudo su -

This would enable him to become a root user.  So root user will have no password, and ftp or sshd would be configured to restrict any root logins.  So any unknown user, say, chris, would login and after logging in would become root, eliminating the need to use root password.
 
Some RPM verification Background:
http://www.rpm.org/max-rpm/ch-rpm-verify.html

SUDO CARTOON
http://ubuntuforums.org/showthread.php?t=1132821

SUDO INTRO
http://www.gratisoft.us/sudo/intro.html
0
 

Author Comment

by:kenwardc
Comment Utility
Hi farzanj

OK - I have implemented the SUDO option on the server and it appears to be working OK. I can reboot the server and login using my personal account but cannot login as root. If I want to execute any root commands I have to type SUDO first, then my personal account password, then the command. Thanks for that.

Next my question revolves around VSFTPD. I realise that it is not secure but how do I get all those web designers whose sites are being hosted on this server upload their files without using an FTP package? Can you upload using SSH?

At this point, because I'm scared that whoever has been coming in was using SSH, I've stopped the service on that server for now. I'm the only one who needs it (other than for the FTP uploading of files) so I guess it's not missed right now.

If you could elaborate on how folks would use SSH to upload their files, that would be super.

Thanks again - I'm feeling much less worried about this server now that it's starting to look as though it's going to he "harder" than it was before.

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
Hi Chris,

For sudo you don't need to reboot the server.  For most Linux operations, there should be no need for reboot.  Try login from your user and if you are happy and satisfied, you should remove the root password from /etc/shadow file.  Instead of password hash, put "!!"

If you are inviting files from unknown users, you have to use ftp so that everyone uses anonymous ftp and uploads files.  If you know the users, who they are and you created a custom account for each one, then why not have them upload files through ssh or https or whatever secure service.

See, telnet and ftp is a pair, getting console and transferring files.  Likewise ssh and scp/sftp makes a pair.  Similarly, you can still put them into chroot jail.  Here is one link to start looking
http://www.howtoforge.com/chrooted-ssh-sftp-tutorial-debian-lenny
When you successfully implement/test it, tell your users to start uploading files using sftp/scp.

One more thing.  Use simple iptables rules to block the ports of your server that you don't use.  You will have to be careful though.  But unmonitored open ports may be exploted.

0
 

Author Closing Comment

by:kenwardc
Comment Utility
Thanks very much for the ongoing help with this one. I'll research the wrappers and close as many ports as possible in an effort to harden further.

I don't know how to use SELinux but will begin research on that and who knows....?

Regards
Chris
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
You are most welcome!  Very few people know SELinux but it is extremely beneficial.  Makes system extremely hard to break.
0
 

Author Comment

by:kenwardc
Comment Utility
I think I spoke too soon about being concerned... I've just been back on the server and the /tmp/aloha directory is back! I'm gutted - I thought the SUDO thing would have stopped that, but obviously this person is getting in via some other method and transferring the files back to the server anytime he wants to.

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
Is ftp still running?
Did you block other unimportant ports?
There may be a script running on your service that is doing it to drive you nuts.
Is there any cron job that is doing that?

Do
cat /var/spool/cron/*

Or
cat /var/spool/cron*/*/*
Whichever works.

Don't get frustrated.  Security needs a lot of patience.
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
You can certainly open a new more specific question after your research
0
 

Author Comment

by:kenwardc
Comment Utility
Hi farzanj

Should I open another question for this ongoing problem? Seems a little unfair to continue here now that this one's closed?

I've just stopped the vsftpd service and a look through the running processes doesn't look as though there are any other ftp services running at this time.

I'll do the port examination thing now and see what I can block or close.

I'll ask a related question and begin with this post. Please would you look out for it?

Thanks for your help.
Chris
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
It would be better to open a question for each area where you need help.  That way you would find more many responses.
0
 

Author Comment

by:kenwardc
Comment Utility
Done! Thanks, farzanj
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

In Solr 4.0 it is possible to atomically (or partially) update individual fields in a document. This article will show the operations possible for atomic updating as well as setting up your Solr instance to be able to perform the actions. One major …
Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now