Solved

Something uploading a VOIP hack to my /tmp folder

Posted on 2011-02-16
57
703 Views
Last Modified: 2012-05-11
With the much appreciated help of one of the experts on this fantastic resource, I have succeeded in hardening my CentOS 5.5 root access by using SUDO. However, someone or something is continually uploading a file called test.tgz to my system, unpacking it and running a VOIP hack called ALOHA. I delete the /tmp/aloha folder and all files, including the test.tgz file but within hours it's back again!

I know I could rebuild the server and use rsync to copy all the websites across to the new server. I know I could migrate all the databases on the server to the new server but there is absolutely nothing stopping this person from coming onto the new server and doing it all over again. Because the server is a production server, I cannot rename it or re-IP it.

A step by step tutorial or help getting either this server hardened to the point where this cannot happen again or, if I build a new server, how to ensure this sort of thing can't happen there, would be sincerely appreciated.

Kind regards
Chris
0
Comment
Question by:kenwardc
  • 27
  • 26
  • 4
57 Comments
 
LVL 31

Accepted Solution

by:
farzanj earned 350 total points
ID: 34908201
Your system might already have "infection" on it.

Did you check your cron?
Check the date and time of the said folders/files that are created.

Do
ls -ld /tmp/aloha

stat /tmp/aloha
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34908226
You have to see if there it creates it with regular intervals.

Check cron jobs for all users
Those jobs are in /var/log/cron/*

See all the jobs
cat /var/log/cron/*

Do you see any job that should not be there?
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34908376
Sorry, I meant
cat /var/spool/cron/*
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34908405
I also want to see the permissions of your /tmp

Do this:
ls -ld /tmp
0
 

Author Comment

by:kenwardc
ID: 34909357
Hi farzanj

Good to see you here! ;)

You wrote:
Did you check your cron?

Yes. There are no cron jobs which are not supposed to be there.

Check the date and time of the said folders/files that are created.
Do
ls -ld /tmp/aloha
stat /tmp/aloha


I'm afraid I can't do that until this person puts the file back there as I've already deleted the file (test.tgz) and the folder (/tmp/aloha) from the /tmp folder. But I'm pretty sure he's going to put it back there again, so will do that the next time and get back to you with the results.

I also want to see the permissions of your /tmp
Do this:
ls -ld /tmp


drwxrwxrwt 21 root root 4096 Feb 16 18:13 /tmp

Looking forward to your reply!

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34909559
Is there a specific reason you want to keep the /tmp permissions wide open for the world.  Can you make is restricted, like

chmod 1700 /tmp

This would at least tell if the hacker can still create files or not
0
 
LVL 7

Assisted Solution

by:namol
namol earned 150 total points
ID: 34909835
Since it's appearing in the /tmp directory and you mentioned various websites running on this server, is anyone running any type of upload application on any of these websites? A photo uploader, a web gallery etc. It might be someone uploading a .tgz file and then trying to exploit the server via the web interface.
0
 

Author Comment

by:kenwardc
ID: 34911126
@farzanj: What effect on running websites etc. would the CHMOD 1700 have? If someone was uploading to their website via FTP it would go into the website directory which they are chrooted into. I'm happy to make the change and watch to see what happens but don't want to disrupt the sites running on the server.

@namol: I'll have a look. At this point not sure what folks are using their sites for. Is there a script or application I can run to search for this sort of application? Another thought - is there a utility I could use to search for vulnerable php forms running on the various sites on the server?

Regards
Chris
0
 

Author Comment

by:kenwardc
ID: 34911345
Out of interest, there is a .htaccess file in the /tmp folder which I believe was put there because some of the folks with Joomla sites needed it. It reads like this:

SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006
Options None
Options +FollowSymLinks

Does this look OK or is it something we don't want in /tmp ??
0
 

Author Comment

by:kenwardc
ID: 34911373
Sorry - I mean Drupal!
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34912964
I meant that /tmp should not be world writable.  It is a temporary directory that is deleted when system reboots, unless we are talking about two different things.  

FollowSymLinks is dangerous.   A symlink may point to some very destructive code.

It is very much possible that the hacker is using multiple means to hack.  For instance he may put some malicious code on your system by ftp (which is easy) and then somehow make is executed through some symlink.
0
 

Author Comment

by:kenwardc
ID: 34915090
Hi farzanj

Is the fact that there is a symlink identified by the 't' at the end of the permissions?
How do I tell where the link is going to on the system?
What are the current permissions (in case I need to chmod it back to what it is)?
Will the chmod 1700 kill the symlink?

Questions, questions, questions? How do you put up with us newbies!? ;)

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34915611
Hi Chris,

I am only talking about practices, "may be's" and "could be's".  Finding hacker's tricks can be very time consuming and frustrating.  I am not even sitting on the system, I cannot give you any verdict.  My intention is to make your system hard enough to protect you from the evil.  I have heard that some hackers are so good they can pass through three firewall -- just a gossip, I don't know it is true or not.

Back to your question:
1.  No, t shows -> executable and sticky bit permission.  It means that any one can enter the folder (any one in the world) and if it is on a file it would mean that any one in the world can execute a program on the system.  The sticky bit part means that only the creator of a file can erase it.  In other words, if you do ls -l and in the user name it says, john, then user mike cannot delete the file despite having the right to write in the folder.
2.  I don't know.  But you can find all the symlinks on the system or on a folder if you want by issuing the following command
find / -type l
You can  put any path instead of / and l stands for link.
3.  You should have restrictive permissions.  Normally the last three bits should not be set unless there is a good reason.  In other words, the "others" part of the premissions should not be set at all, or if you do, you there should be a reason and you should know what you are doing.  For /tmp, I did not think any group or 'others' should have had permission to do anything in /tmp
4.  1 is for sticky bit (keeping it the way it was and 00 for removing all permissions from users and others.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34915616
Last line from groups and others  --> instead of users and others: Sorry for the typo :)
0
 

Author Comment

by:kenwardc
ID: 34915779
Hi again

OK - I've done the chmod 1700 but I didn't do a -R was that OK? The directory now looks like this:

 tmp after chmod 1700
I see that some folders for the Gnome stuff are still world writable like the other stuff was. Should I have done the chmod with the -R option?

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34915809
Are you keeping any important files on /tmp?  If you are, move somewhere else and remove every thing from /tmp.  Important files should not be residing on /tmp.  I am saying because I am assuming that your /tmp is the same old /tmp that exists on every system not some special /tmp that you created yourself
0
 

Author Comment

by:kenwardc
ID: 34916199
Hi farzanj

No - /tmp is just the same old /tmp that was created when I installed the box. Apart from the Gnome desktop files there, I don't every manually upload anything in there.

Does the folder look OK to you?

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34916529
Based on your input that the mystery files were getting generated there, I presume that /tmp has been the target of a hacker.
It would be my first instinct to remove privileges from that folder.  Since you make permissions : 1700, I think no one can access it by ordinary means now.  If I were you, I would still get rid of anything in there.  Normally no important files are kept in there anyway.

This is the first step.

Second:
You should harden your Apache.
http://xianshield.org/guides/apache2.0guide.html

0
 

Author Comment

by:kenwardc
ID: 34917524
Hi fanzanj

I have a problem.... one of my web-host customers has a website on this server and has just complained that none of his php forms are working. This is obviously caused by the chmod 1700. Can you think of any reason why this is happening and how to rectify it without returning to the original permissions?

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34917643
Oops.  I honestly did not think anything would depend on /tmp directory.  Try reverting the permissions and see if it fixes that problem.
0
 

Author Comment

by:kenwardc
ID: 34917670
I've changed the permissions to chmod 1757 in an effort to see if that's not quite as bad as before when it was completely open. I'll let you know what the customer says?

Cheers, farzanj

Chris
0
 

Author Comment

by:kenwardc
ID: 34918513
Customer's CMS backend now working again with the new permissions. Not sure how secure the /tmp folder is now with 1757 but so far no repeat performance of the hack.... I'm on my way out now so will take another look at it later when I return to the office.

Thanks as always for your ongoing help with this, farzanj

Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34918729
It is not secure at all.  The last 7 gives everyone full permission.  What you can do is this:  See the customer's php script's ownerships.  Are they residing on your server?
Make a customer group.  Change group of /tmp to customer group and then allow full group permissions but no other permissions.

Method II:  Enable ACLs.  Then give customers access to /tmp and no one else.
0
 

Author Comment

by:kenwardc
ID: 34922171
Hi farzan

I'm afraid you're talking way above my head. I'm slowly getting to grips with the OS but ACLs for the /tmp folder .... OOPS!? ;)

What I've now done is follow the link I found HERE and have followed it to secure the /tmp and /var/tmp folders. The only difference is that I've used 1755 rather than 1700 because 1700 kills the CMS package we were talking about earlier.

So... we're another step closer? The file has not been uploaded today at all. Fingers crossed!

Right - I'm off to read the link to harden Apache that you gave me above! I'll give you some feedback as soon as I've digested it.

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34922346
Is /tmp on a separate partition?

It is easy to do ACLs.  Not too much problem.
0
 

Author Comment

by:kenwardc
ID: 34922953
Hi farzanj

A little problem which you may be able to help with? The mysql server seems to require that the /tmp folder has world writable permissions. If I change the permissions on the /tmp folder to 1755 for example, then mysql stops working properly with an error about not being able to write to the /tmp/$sql folder. As soon as I chmod to something like 1757 then mysql works fine. Surely there must be a way to get mysql to work without forcing world writable /tmp ?

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34923058
Hi Chris,

You need flexibility.  Use ACLs.

do this
df -hT /tmp

What is the partition name for /tmp.

In /etc/fstab, use acl option

Remount
mount /tmp -o remount,rw,acl

Then you are ready to allow and deny.
setfacl -m u:client:rxw /tmp
setfacl -m u:mysql:rwx /tmp
setfacl -x d:o /tmp
0
 

Author Comment

by:kenwardc
ID: 34924570
Hi farzanj

Thanks for the ACL suggestion. I'm not really sure how to implement this so will do some research for later. In the meantime I have another query about one of the things I've done to secure the /tmp directory.

I followed the link IN THIS URL to secure /tmp.

When completed, and I did a df -kh, I got a list of all including the new /tmp mounted file. Now, however, when I do a df -kh, I only get the following:

 DF -KH after /tmp mount
Have I screwed up or is this normal and we should no longer see the /tmp mounted entry?

Cheers
Chris
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 31

Expert Comment

by:farzanj
ID: 34925892
Did you reboot?  Can you still reach /tmp?  Try reading something from it.
0
 
LVL 7

Expert Comment

by:namol
ID: 34929100
What does your /etc/fstab say?

If you didn't add the mount point to fstab, anytime you reboot it will "disappear".
0
 

Author Comment

by:kenwardc
ID: 34935238
@farzanj: Yes - I can read from it etc. and it's there for a LS dir listing but doesn't appear in the DF -KH

@namol: fstab screen shot below. Let me know if it's OK?

 The /etc/fstab file with /tmp mounted as 1GB disk
0
 

Author Comment

by:kenwardc
ID: 34936846
Hi folks

Here is a screenshot of the /tmp folder this morning. The ALOHA directory is back as is the test.tgz file.

How on earth am I going to stop this?

Cheers
Chris

The hack is back!
0
 

Author Comment

by:kenwardc
ID: 34936853
Problem is that if I take away from write access from "world" it seems that the mySQL installation fails miserably. Can't even load phpMyAdmin to browse a database without the settings in /tmp. Is there some way perhaps to force mysqld to use another folder, so I can secure /tmp?

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34937812
You got the answer.

I don't remember seeing the "ls -l" view of aloha.  It is being written by user and group apache.

If you block apache from accessing /tmp, how would that affect you??  You can block it using ACLs.  I just told you all the ACL commands above you needed to do.

df command gives file systems not file.  You made /tmp a file on /dev, which resides on / partition.

0
 
LVL 31

Expert Comment

by:farzanj
ID: 34937828
Regarding mysql, I don't know how you are installing it.  If you are installing by source, you have prefix option and other flags that you can use customize your installation completely.  Similarly RPM installation can be configured too.

Using ACLs you can allow mySQL user to use /tmp and disallow apache user to use it.
0
 

Author Comment

by:kenwardc
ID: 34938136
Hi farzanj

Not sure how denying apache user from /tmp would affect the system. Also not really clear on how the ACLs work which is why I haven't tried it yet. Could you be more specific about the commands to use to display all disks on the system?

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34938887
Answer this question:   What was the user name and group name of the user who created this file?
It is set as apache.  It is very unlikely that user  had some other credentials and then used chown to change it to apache. It is unlikely (or more advanced) if the hacker is changing it later.

The hacker, as it appears, is using the apache.  This also bring another apparent configuration problem to my attention.  The user apache has a shell.  Typically, it should not have a shell and should not have a password, should not have a home directory.  In the /etc/passwd file he should have /bin/false (or /sbin/nologin) instead of /bin/bash etc.  If you want to try this first for a day or two, go ahead.
0
 

Author Comment

by:kenwardc
ID: 34939377
Thanks, farzanj

I've attached a screenshot of the user 'Apache' for you so you can see that this user has no login to shell and password is locked.

Cheers
Chris
 user APACHE
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34942450
In the ongoing investigation, I would like user and group apache to be kept from writing on /tmp.  I want to see whether the files are still created and if so, with what user and group.  And ACL is, in your case, the only way to do this.

It is simple as described above.  Remount your partition with acl option.  The use getfacl command to see ACLs and setfacl to set acls.  I believe there is a GUI tool to do that too if you feel that is easier.
0
 

Author Comment

by:kenwardc
ID: 34965233
Hi farzanj

If I do df -hT /tmp I get the following:

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3     49G   25G   22G  54% /

Which doesn't look right to me.

The /etc/fstab file looks like this:
LABEL=/               /                  ext3     defaults                      1 1
tmpfs                    /dev/shm     tmpfs   defaults                       0 0
devpts                  /dev/pts       devpts  gid=5,mode=620          0 0
sysfs                    /sys            sysfs    defaults                       0 0
proc                     /proc            proc      defaults                       0 0
/dev/tmpDIR         /tmp             ext3      loop,nosuid,noexec,rw  0 0
LABEL=SWAP-sda2                swap     swap    defaults            0 0

I thought that the /tmp folder should mount as a drive using the above, automatically? Have I set this up incorrectly?

Cheers
Chris
0
 
LVL 7

Expert Comment

by:namol
ID: 34970144
This sounds oddly similar to your issue Ken, http://mattiasgeniar.be/2010/12/01/tracking-root-cause-of-apache-spawned-processes-shell-scripts-daemons/ . Do you by chance run cpanel and phpMyadmin on this machine?

Also you asked about tools to use to scan with, I recommend Nessus, by far one of the most encompassing security scanners.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 34970505
Hi Chris,
To the best of my understanding, /tmp does not qualify as a partition.  It is a file created by dd command.  Likewise you also have other device files mounted.  Issue mount command and you may see what you want to see, not sure.  Did you get any where closer to resolve the main security breach issue?
0
 

Author Comment

by:kenwardc
ID: 34989675
@namol
Had a look at Nessus - wow - it's too rich for me. We're a very small ISP - there is no way we could afford to spend that sort of money. Looks like a great product - pity about the price!

@farzanj
Nope - been busy researching the ACL thing and trying to figure out why, when I've apparently mounted /tmp as a disk device, then secured it with the nosuid etc. options, it doesn't come up in the list. I'm lost!
0
 

Author Comment

by:kenwardc
ID: 35048330
Update:
======

I'm still not able to stop this person from re-loading the Aloha application in the /tmp folder and the file test.tgz is still appearing in there on a daily basis. I haven't yet learned enough about ACLs to protect the /tmp folder so not sure where to go to from here...

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35048435
If you want to do it, it is easy.  If you don't want to do it, that is fine too.  But you need to issue the following two commands only.

 
mount -o remount,rw,acl /dev/tmpDIR  /tmp
getfacl -R /tmp

Open in new window

0
 

Author Comment

by:kenwardc
ID: 35048832
Hi farzan

Can you tell me what effect that exact two commands will have on the server and whether it might affect anything that is currently using the /tmp folder legitimately?

Sorry to be so paranoid but it is a live server and I'm worried about the consequences of my actions.

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35049133
The first command remounts with acl option.  It should not have any effect.  Second one is to see the current state of acl's.  Both combined should only enable ACLs with NO effect to your server.  It would not make ACLs persistent after reboot.  For that you will have to modify /etc/fstab

Once this is successful, we can proceed.
0
 

Author Comment

by:kenwardc
ID: 35049532
Hi Farzan

When running the first command on CentOS I got an error:

mount: you must specify the filesystem type

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35049992
Try
 
mount -t ext3 -o remount,rw,acl /dev/tmpDIR  /tmp

Open in new window

0
 

Author Comment

by:kenwardc
ID: 35054210
Hi farzanj

Now I get: mount -t ext3 -o remount,rw,acl /dev/tmpDIR /tmp

mount: /tmp not mounted already, or bad option
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35055658
Well this command was out of your /etc/fstab entry.  Idea is to remount it with acl option.
0
 
LVL 7

Expert Comment

by:namol
ID: 35151411
Chris: Check out OpenVAS, it's a Nessus 3x fork from before Nessus went closed source. http://www.openvas.org/index.html It works with all the same rules etc.
0
 

Author Comment

by:kenwardc
ID: 35152607
Thanks, namol will have a look at it now.

@farzanj: Sorry about the silence. Will get to the ACLs shortly. So much to do, so little time! ;)

Thanks, guys!
Chris
0
 

Author Comment

by:kenwardc
ID: 35241711
Hi Folks

As I don't have time at the moment to go off and learn all about ACLs and how to CHROOT Apache, I've decided to close this question and award the points for the wonderful amount of help and patience you've show me. I think this question and the answers you have provided will be of great value to others in my shoes, so it'll stay as a monument to your expertise and patience.

THANK YOU.

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35241804
Sorry got too busy.

On EE the best way to get help is to open a very narrow question as it would help you solve problems in a matter of hours.  Also it would become a usable part in the knowledge base.
0
 

Author Comment

by:kenwardc
ID: 35242093
Super - thanks, FarzanJ - looking forward to coming across you again sometime. Take care.

Cheers
Chris
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35242277
Welcome. At your service.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

This article summarizes using a simple matrix to map the different type of phishing attempts and its targeted victims. It also run through many scam scheme scenario with "real" phished emails. There are safeguards highlighted to stay vigilance and h…
You cannot be 100% sure that you can protect your organization against crypto ransomware but you can lower down the risk and impact of the infection.
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now