Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 714
  • Last Modified:

Something uploading a VOIP hack to my /tmp folder

With the much appreciated help of one of the experts on this fantastic resource, I have succeeded in hardening my CentOS 5.5 root access by using SUDO. However, someone or something is continually uploading a file called test.tgz to my system, unpacking it and running a VOIP hack called ALOHA. I delete the /tmp/aloha folder and all files, including the test.tgz file but within hours it's back again!

I know I could rebuild the server and use rsync to copy all the websites across to the new server. I know I could migrate all the databases on the server to the new server but there is absolutely nothing stopping this person from coming onto the new server and doing it all over again. Because the server is a production server, I cannot rename it or re-IP it.

A step by step tutorial or help getting either this server hardened to the point where this cannot happen again or, if I build a new server, how to ensure this sort of thing can't happen there, would be sincerely appreciated.

Kind regards
Chris
0
kenwardc
Asked:
kenwardc
  • 27
  • 26
  • 4
2 Solutions
 
farzanjCommented:
Your system might already have "infection" on it.

Did you check your cron?
Check the date and time of the said folders/files that are created.

Do
ls -ld /tmp/aloha

stat /tmp/aloha
0
 
farzanjCommented:
You have to see if there it creates it with regular intervals.

Check cron jobs for all users
Those jobs are in /var/log/cron/*

See all the jobs
cat /var/log/cron/*

Do you see any job that should not be there?
0
 
farzanjCommented:
Sorry, I meant
cat /var/spool/cron/*
0
Building an Effective Phishing Protection Program

Join Director of Product Management Todd OBoyle on April 26th as he covers the key elements of a phishing protection program. Whether you’re an old hat at phishing education or considering starting a program -- we'll discuss critical components that should be in any program.

 
farzanjCommented:
I also want to see the permissions of your /tmp

Do this:
ls -ld /tmp
0
 
kenwardcAuthor Commented:
Hi farzanj

Good to see you here! ;)

You wrote:
Did you check your cron?

Yes. There are no cron jobs which are not supposed to be there.

Check the date and time of the said folders/files that are created.
Do
ls -ld /tmp/aloha
stat /tmp/aloha


I'm afraid I can't do that until this person puts the file back there as I've already deleted the file (test.tgz) and the folder (/tmp/aloha) from the /tmp folder. But I'm pretty sure he's going to put it back there again, so will do that the next time and get back to you with the results.

I also want to see the permissions of your /tmp
Do this:
ls -ld /tmp


drwxrwxrwt 21 root root 4096 Feb 16 18:13 /tmp

Looking forward to your reply!

Cheers
Chris
0
 
farzanjCommented:
Is there a specific reason you want to keep the /tmp permissions wide open for the world.  Can you make is restricted, like

chmod 1700 /tmp

This would at least tell if the hacker can still create files or not
0
 
namolCommented:
Since it's appearing in the /tmp directory and you mentioned various websites running on this server, is anyone running any type of upload application on any of these websites? A photo uploader, a web gallery etc. It might be someone uploading a .tgz file and then trying to exploit the server via the web interface.
0
 
kenwardcAuthor Commented:
@farzanj: What effect on running websites etc. would the CHMOD 1700 have? If someone was uploading to their website via FTP it would go into the website directory which they are chrooted into. I'm happy to make the change and watch to see what happens but don't want to disrupt the sites running on the server.

@namol: I'll have a look. At this point not sure what folks are using their sites for. Is there a script or application I can run to search for this sort of application? Another thought - is there a utility I could use to search for vulnerable php forms running on the various sites on the server?

Regards
Chris
0
 
kenwardcAuthor Commented:
Out of interest, there is a .htaccess file in the /tmp folder which I believe was put there because some of the folks with Joomla sites needed it. It reads like this:

SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006
Options None
Options +FollowSymLinks

Does this look OK or is it something we don't want in /tmp ??
0
 
kenwardcAuthor Commented:
Sorry - I mean Drupal!
0
 
farzanjCommented:
I meant that /tmp should not be world writable.  It is a temporary directory that is deleted when system reboots, unless we are talking about two different things.  

FollowSymLinks is dangerous.   A symlink may point to some very destructive code.

It is very much possible that the hacker is using multiple means to hack.  For instance he may put some malicious code on your system by ftp (which is easy) and then somehow make is executed through some symlink.
0
 
kenwardcAuthor Commented:
Hi farzanj

Is the fact that there is a symlink identified by the 't' at the end of the permissions?
How do I tell where the link is going to on the system?
What are the current permissions (in case I need to chmod it back to what it is)?
Will the chmod 1700 kill the symlink?

Questions, questions, questions? How do you put up with us newbies!? ;)

Cheers
Chris
0
 
farzanjCommented:
Hi Chris,

I am only talking about practices, "may be's" and "could be's".  Finding hacker's tricks can be very time consuming and frustrating.  I am not even sitting on the system, I cannot give you any verdict.  My intention is to make your system hard enough to protect you from the evil.  I have heard that some hackers are so good they can pass through three firewall -- just a gossip, I don't know it is true or not.

Back to your question:
1.  No, t shows -> executable and sticky bit permission.  It means that any one can enter the folder (any one in the world) and if it is on a file it would mean that any one in the world can execute a program on the system.  The sticky bit part means that only the creator of a file can erase it.  In other words, if you do ls -l and in the user name it says, john, then user mike cannot delete the file despite having the right to write in the folder.
2.  I don't know.  But you can find all the symlinks on the system or on a folder if you want by issuing the following command
find / -type l
You can  put any path instead of / and l stands for link.
3.  You should have restrictive permissions.  Normally the last three bits should not be set unless there is a good reason.  In other words, the "others" part of the premissions should not be set at all, or if you do, you there should be a reason and you should know what you are doing.  For /tmp, I did not think any group or 'others' should have had permission to do anything in /tmp
4.  1 is for sticky bit (keeping it the way it was and 00 for removing all permissions from users and others.
0
 
farzanjCommented:
Last line from groups and others  --> instead of users and others: Sorry for the typo :)
0
 
kenwardcAuthor Commented:
Hi again

OK - I've done the chmod 1700 but I didn't do a -R was that OK? The directory now looks like this:

 tmp after chmod 1700
I see that some folders for the Gnome stuff are still world writable like the other stuff was. Should I have done the chmod with the -R option?

Cheers
Chris
0
 
farzanjCommented:
Are you keeping any important files on /tmp?  If you are, move somewhere else and remove every thing from /tmp.  Important files should not be residing on /tmp.  I am saying because I am assuming that your /tmp is the same old /tmp that exists on every system not some special /tmp that you created yourself
0
 
kenwardcAuthor Commented:
Hi farzanj

No - /tmp is just the same old /tmp that was created when I installed the box. Apart from the Gnome desktop files there, I don't every manually upload anything in there.

Does the folder look OK to you?

Cheers
Chris
0
 
farzanjCommented:
Based on your input that the mystery files were getting generated there, I presume that /tmp has been the target of a hacker.
It would be my first instinct to remove privileges from that folder.  Since you make permissions : 1700, I think no one can access it by ordinary means now.  If I were you, I would still get rid of anything in there.  Normally no important files are kept in there anyway.

This is the first step.

Second:
You should harden your Apache.
http://xianshield.org/guides/apache2.0guide.html

0
 
kenwardcAuthor Commented:
Hi fanzanj

I have a problem.... one of my web-host customers has a website on this server and has just complained that none of his php forms are working. This is obviously caused by the chmod 1700. Can you think of any reason why this is happening and how to rectify it without returning to the original permissions?

Cheers
Chris
0
 
farzanjCommented:
Oops.  I honestly did not think anything would depend on /tmp directory.  Try reverting the permissions and see if it fixes that problem.
0
 
kenwardcAuthor Commented:
I've changed the permissions to chmod 1757 in an effort to see if that's not quite as bad as before when it was completely open. I'll let you know what the customer says?

Cheers, farzanj

Chris
0
 
kenwardcAuthor Commented:
Customer's CMS backend now working again with the new permissions. Not sure how secure the /tmp folder is now with 1757 but so far no repeat performance of the hack.... I'm on my way out now so will take another look at it later when I return to the office.

Thanks as always for your ongoing help with this, farzanj

Chris
0
 
farzanjCommented:
It is not secure at all.  The last 7 gives everyone full permission.  What you can do is this:  See the customer's php script's ownerships.  Are they residing on your server?
Make a customer group.  Change group of /tmp to customer group and then allow full group permissions but no other permissions.

Method II:  Enable ACLs.  Then give customers access to /tmp and no one else.
0
 
kenwardcAuthor Commented:
Hi farzan

I'm afraid you're talking way above my head. I'm slowly getting to grips with the OS but ACLs for the /tmp folder .... OOPS!? ;)

What I've now done is follow the link I found HERE and have followed it to secure the /tmp and /var/tmp folders. The only difference is that I've used 1755 rather than 1700 because 1700 kills the CMS package we were talking about earlier.

So... we're another step closer? The file has not been uploaded today at all. Fingers crossed!

Right - I'm off to read the link to harden Apache that you gave me above! I'll give you some feedback as soon as I've digested it.

Cheers
Chris
0
 
farzanjCommented:
Is /tmp on a separate partition?

It is easy to do ACLs.  Not too much problem.
0
 
kenwardcAuthor Commented:
Hi farzanj

A little problem which you may be able to help with? The mysql server seems to require that the /tmp folder has world writable permissions. If I change the permissions on the /tmp folder to 1755 for example, then mysql stops working properly with an error about not being able to write to the /tmp/$sql folder. As soon as I chmod to something like 1757 then mysql works fine. Surely there must be a way to get mysql to work without forcing world writable /tmp ?

Cheers
Chris
0
 
farzanjCommented:
Hi Chris,

You need flexibility.  Use ACLs.

do this
df -hT /tmp

What is the partition name for /tmp.

In /etc/fstab, use acl option

Remount
mount /tmp -o remount,rw,acl

Then you are ready to allow and deny.
setfacl -m u:client:rxw /tmp
setfacl -m u:mysql:rwx /tmp
setfacl -x d:o /tmp
0
 
kenwardcAuthor Commented:
Hi farzanj

Thanks for the ACL suggestion. I'm not really sure how to implement this so will do some research for later. In the meantime I have another query about one of the things I've done to secure the /tmp directory.

I followed the link IN THIS URL to secure /tmp.

When completed, and I did a df -kh, I got a list of all including the new /tmp mounted file. Now, however, when I do a df -kh, I only get the following:

 DF -KH after /tmp mount
Have I screwed up or is this normal and we should no longer see the /tmp mounted entry?

Cheers
Chris
0
 
farzanjCommented:
Did you reboot?  Can you still reach /tmp?  Try reading something from it.
0
 
namolCommented:
What does your /etc/fstab say?

If you didn't add the mount point to fstab, anytime you reboot it will "disappear".
0
 
kenwardcAuthor Commented:
@farzanj: Yes - I can read from it etc. and it's there for a LS dir listing but doesn't appear in the DF -KH

@namol: fstab screen shot below. Let me know if it's OK?

 The /etc/fstab file with /tmp mounted as 1GB disk
0
 
kenwardcAuthor Commented:
Hi folks

Here is a screenshot of the /tmp folder this morning. The ALOHA directory is back as is the test.tgz file.

How on earth am I going to stop this?

Cheers
Chris

The hack is back!
0
 
kenwardcAuthor Commented:
Problem is that if I take away from write access from "world" it seems that the mySQL installation fails miserably. Can't even load phpMyAdmin to browse a database without the settings in /tmp. Is there some way perhaps to force mysqld to use another folder, so I can secure /tmp?

Cheers
Chris
0
 
farzanjCommented:
You got the answer.

I don't remember seeing the "ls -l" view of aloha.  It is being written by user and group apache.

If you block apache from accessing /tmp, how would that affect you??  You can block it using ACLs.  I just told you all the ACL commands above you needed to do.

df command gives file systems not file.  You made /tmp a file on /dev, which resides on / partition.

0
 
farzanjCommented:
Regarding mysql, I don't know how you are installing it.  If you are installing by source, you have prefix option and other flags that you can use customize your installation completely.  Similarly RPM installation can be configured too.

Using ACLs you can allow mySQL user to use /tmp and disallow apache user to use it.
0
 
kenwardcAuthor Commented:
Hi farzanj

Not sure how denying apache user from /tmp would affect the system. Also not really clear on how the ACLs work which is why I haven't tried it yet. Could you be more specific about the commands to use to display all disks on the system?

Cheers
Chris
0
 
farzanjCommented:
Answer this question:   What was the user name and group name of the user who created this file?
It is set as apache.  It is very unlikely that user  had some other credentials and then used chown to change it to apache. It is unlikely (or more advanced) if the hacker is changing it later.

The hacker, as it appears, is using the apache.  This also bring another apparent configuration problem to my attention.  The user apache has a shell.  Typically, it should not have a shell and should not have a password, should not have a home directory.  In the /etc/passwd file he should have /bin/false (or /sbin/nologin) instead of /bin/bash etc.  If you want to try this first for a day or two, go ahead.
0
 
kenwardcAuthor Commented:
Thanks, farzanj

I've attached a screenshot of the user 'Apache' for you so you can see that this user has no login to shell and password is locked.

Cheers
Chris
 user APACHE
0
 
farzanjCommented:
In the ongoing investigation, I would like user and group apache to be kept from writing on /tmp.  I want to see whether the files are still created and if so, with what user and group.  And ACL is, in your case, the only way to do this.

It is simple as described above.  Remount your partition with acl option.  The use getfacl command to see ACLs and setfacl to set acls.  I believe there is a GUI tool to do that too if you feel that is easier.
0
 
kenwardcAuthor Commented:
Hi farzanj

If I do df -hT /tmp I get the following:

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3     49G   25G   22G  54% /

Which doesn't look right to me.

The /etc/fstab file looks like this:
LABEL=/               /                  ext3     defaults                      1 1
tmpfs                    /dev/shm     tmpfs   defaults                       0 0
devpts                  /dev/pts       devpts  gid=5,mode=620          0 0
sysfs                    /sys            sysfs    defaults                       0 0
proc                     /proc            proc      defaults                       0 0
/dev/tmpDIR         /tmp             ext3      loop,nosuid,noexec,rw  0 0
LABEL=SWAP-sda2                swap     swap    defaults            0 0

I thought that the /tmp folder should mount as a drive using the above, automatically? Have I set this up incorrectly?

Cheers
Chris
0
 
namolCommented:
This sounds oddly similar to your issue Ken, http://mattiasgeniar.be/2010/12/01/tracking-root-cause-of-apache-spawned-processes-shell-scripts-daemons/ . Do you by chance run cpanel and phpMyadmin on this machine?

Also you asked about tools to use to scan with, I recommend Nessus, by far one of the most encompassing security scanners.
0
 
farzanjCommented:
Hi Chris,
To the best of my understanding, /tmp does not qualify as a partition.  It is a file created by dd command.  Likewise you also have other device files mounted.  Issue mount command and you may see what you want to see, not sure.  Did you get any where closer to resolve the main security breach issue?
0
 
kenwardcAuthor Commented:
@namol
Had a look at Nessus - wow - it's too rich for me. We're a very small ISP - there is no way we could afford to spend that sort of money. Looks like a great product - pity about the price!

@farzanj
Nope - been busy researching the ACL thing and trying to figure out why, when I've apparently mounted /tmp as a disk device, then secured it with the nosuid etc. options, it doesn't come up in the list. I'm lost!
0
 
kenwardcAuthor Commented:
Update:
======

I'm still not able to stop this person from re-loading the Aloha application in the /tmp folder and the file test.tgz is still appearing in there on a daily basis. I haven't yet learned enough about ACLs to protect the /tmp folder so not sure where to go to from here...

Cheers
Chris
0
 
farzanjCommented:
If you want to do it, it is easy.  If you don't want to do it, that is fine too.  But you need to issue the following two commands only.

 
mount -o remount,rw,acl /dev/tmpDIR  /tmp
getfacl -R /tmp

Open in new window

0
 
kenwardcAuthor Commented:
Hi farzan

Can you tell me what effect that exact two commands will have on the server and whether it might affect anything that is currently using the /tmp folder legitimately?

Sorry to be so paranoid but it is a live server and I'm worried about the consequences of my actions.

Cheers
Chris
0
 
farzanjCommented:
The first command remounts with acl option.  It should not have any effect.  Second one is to see the current state of acl's.  Both combined should only enable ACLs with NO effect to your server.  It would not make ACLs persistent after reboot.  For that you will have to modify /etc/fstab

Once this is successful, we can proceed.
0
 
kenwardcAuthor Commented:
Hi Farzan

When running the first command on CentOS I got an error:

mount: you must specify the filesystem type

Cheers
Chris
0
 
farzanjCommented:
Try
 
mount -t ext3 -o remount,rw,acl /dev/tmpDIR  /tmp

Open in new window

0
 
kenwardcAuthor Commented:
Hi farzanj

Now I get: mount -t ext3 -o remount,rw,acl /dev/tmpDIR /tmp

mount: /tmp not mounted already, or bad option
0
 
farzanjCommented:
Well this command was out of your /etc/fstab entry.  Idea is to remount it with acl option.
0
 
namolCommented:
Chris: Check out OpenVAS, it's a Nessus 3x fork from before Nessus went closed source. http://www.openvas.org/index.html It works with all the same rules etc.
0
 
kenwardcAuthor Commented:
Thanks, namol will have a look at it now.

@farzanj: Sorry about the silence. Will get to the ACLs shortly. So much to do, so little time! ;)

Thanks, guys!
Chris
0
 
kenwardcAuthor Commented:
Hi Folks

As I don't have time at the moment to go off and learn all about ACLs and how to CHROOT Apache, I've decided to close this question and award the points for the wonderful amount of help and patience you've show me. I think this question and the answers you have provided will be of great value to others in my shoes, so it'll stay as a monument to your expertise and patience.

THANK YOU.

Cheers
Chris
0
 
farzanjCommented:
Sorry got too busy.

On EE the best way to get help is to open a very narrow question as it would help you solve problems in a matter of hours.  Also it would become a usable part in the knowledge base.
0
 
kenwardcAuthor Commented:
Super - thanks, FarzanJ - looking forward to coming across you again sometime. Take care.

Cheers
Chris
0
 
farzanjCommented:
Welcome. At your service.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

NEW Internet Security Report Now Available!

WatchGuard’s Threat Lab is a group of dedicated threat researchers committed to helping you stay ahead of the bad guys by providing in-depth analysis of the top security threats to your network.  Check out this quarters report on the threats that shook the industry in Q4 2017.

  • 27
  • 26
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now