Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 738
  • Last Modified:

Linux Server Compromised - now I can't delete or rename (2) files

I recently had a client have their Linux box compromised due to a globally open SSH port and a weak root password.

After the compromise, a CRON job was left running every minute that started a process that logged the server into an IRC channel in order to become part of a BOTNET or some other such situation.  Regardless, I was able to kill all processes related to this by killing the CRON process and all of the processes that were spawned by this.

I have (2) things left to clean up: the CRONTAB file still holds entries in it for starting a specific process named "/bin/f" , and I am UNABLE to modify the CRONTAB file (access is denied) and I am UNABLE to delete / rename the file "/bin/f".  The files permissions are as follows:

/etc/crontab:
-rw-r--r--   1 root root     283 Apr 17 01:59 crontab

/bin/f:
-rwxr-xr-x  1 root root  885401 Feb 18 01:19 f

I have tried to CHMOD the files, and I get "changing permissions of "f": Operation not permitted.
I have tried to CHOWN the files, and I get the same thing:  Operation not permitted.

Can someone tell me what I have to do to get rid of these files?  I have looked at processes running with ps -aux and I don't see anything running "/bin/f" anymore, and there are no crontab processes running, so I don't think that "/etc/crontab" is being held open.  

All of the volumes are ext3 file systems, here is the output of mount:

/dev/sda5 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda3 on /var/log type ext3 (rw)
/dev/sda2 on /var/lib/mysql type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)


Thanks!
0
jkeegan123
Asked:
jkeegan123
  • 8
  • 7
  • 3
  • +1
1 Solution
 
JChrisonCommented:
You use sudo and still says access denied ?

try to see if there is any new user has been created.
0
 
jkeegan123Author Commented:
I'm in as root, and cannot make these changes to these files.  I can make changes to ALL other files still.

Yes, I also tried performing these operations using SUDO and the same thing happens.  

It is not my box so I'm not sure if there are new users, but it didn't look like anything new was there.  All of this was done with root and the weak password that it had.

0
 
farzanjCommented:
From tags it appears that it is Red Hat linux.

You have to verify RPMs.

Often in attacks, the binaries are changed.  So the ls is no more a ls -- it would not show some of files the attacher wants to keep.  ps may not be ps as it may not show some of the processes the attacker doesn't want you to see.

In your case rm doesn't appear to be the actual binary.

If the system was not very properly protected, your best approach would be to reinstall the OS.

Short of that, you can start verifying RPMS.  RPM is a very good mechanism as you can verify it and it can tell you what changes were made after a certain time or after the original install.   RPMS are also signed so if anything changes, you can check the signatures.  Any package that doesn't pass those tests should be removed.

 
0
Nothing ever in the clear!

This technical paper will help you implement VMware’s VM encryption as well as implement Veeam encryption which together will achieve the nothing ever in the clear goal. If a bad guy steals VMs, backups or traffic they get nothing.

 
JChrisonCommented:
try this first

sudo chattr -i /bin/f

and then

sudo rm /bin/f



0
 
jkeegan123Author Commented:
I was able to use "kill" to end the cron job that was spawning these processes, and other operations seem to work as expected.  I think this was just a lucky hit from a script kid.  Any ideas on those files?
0
 
jkeegan123Author Commented:
chattr does not appear to be a valid command on this install.
0
 
JChrisonCommented:
I repeate that you should check if these files are immutable.

chattr +i your_file makes the file immutable and even root user cannot delete them

to reverse that you have to run chattr -i your_file
0
 
farzanjCommented:
You should issue this command as noted above:

chattr -i <FILE>

rm -f <FILE>

The problem is that the person also removed a major rpm as well.

Please install this:
yum install e2fsprogs

0
 
jkeegan123Author Commented:
@farzanj:  Will this disrupt any of the running OS (yum install e2fsprogs)?  Would it be easier to just find a copy of "chattr" to use rather than download the entire "e2fsprogs" ?
0
 
farzanjCommented:
It should not disrupt anything.  It has some very basic programs.  In Red Hat, you should always install packages, not files.
0
 
farzanjCommented:
You can verify your packages with rpm verification.

rpm -qa | xargs rpm -V

For more information:
man rpm
0
 
jkeegan123Author Commented:
@farzanj:  When I ran "yum  install e2fsprogs" I got the following error:


Error Message:
        Please run rhn_register as root on this client
Error Class Code: 9
Error Class Info: Invalid System Credentials.
Explanation:
     An error has occurred while processing your request. If this problem
     persists please enter a bug report at bugzilla.redhat.com.
     If you choose to submit the bug report, please be sure to include
     details of what you were trying to do when this error occurred and
     details on how to reproduce this problem.

Setting up Install Process
Parsing package install arguments
Package e2fsprogs-1.39-20.el5.i386 installed and not available
Nothing to do
0
 
farzanjCommented:
The above would tell you the changes that were made to your rpm packages.  You should dump it into a file and carefully review it.  You can ignore reviewing documentation changes with
rpm -qa | xargs rpm -V --excludedocs
0
 
farzanjCommented:
Oops!!

Was your system registered to RHN before the attack.  It may be a little more than just a lucky attack.  The message that you received is a red flag.  Sorry!
0
 
jkeegan123Author Commented:
I am CERTAIN that this particular client was not registered to RHN ever.  You know the type I'm sure.

The results of rpm -qa | xargs rpm -V are:


.M......    /usr/bin/wget
S.5....T  c /etc/crontab
.......T  c /etc/mail/sendmail.cf
S.5....T  c /var/log/mail/statistics
......G.    /var/cache/samba/winbindd_privileged
S.5....T  c /etc/ppp/chap-secrets
S.5....T  c /etc/ppp/pap-secrets
.......T  c /etc/audit/auditd.conf
S.5....T    /usr/share/logwatch/default.conf/logwatch.conf
S.5....T    /usr/lib/qt-3.3/etc/settings/qtrc
S.5....T    /usr/share/icons/hicolor/icon-theme.cache
S.5....T  c /etc/sudoers
S.5....T  c /etc/freetds.conf
S.5....T  c /etc/printcap
SM5....T  c /etc/sysconfig/iptables-config
....L...  c /etc/pam.d/system-auth
..5....T  c /etc/inittab
S.5....T  c /etc/xml/catalog
S.5....T  c /usr/share/sgml/docbook/xmlcatalog
S.5....T  c /etc/php.ini
S.5....T  c /etc/sysconfig/system-config-securitylevel
Unsatisfied dependencies for libtiff-devel-3.8.2-7.el5.i386: libtiff = 3.8.2-7.el5
..5....T  c /usr/lib/security/classpath.security
S.5....T  c /etc/sane.d/dll.conf
S.5....T  c /etc/sysconfig/vncservers
.......T    /usr/share/pear/.depdb
.......T    /usr/share/pear/.depdblock
..5....T    /usr/share/pear/.filemap
.......T    /usr/share/pear/.lock
missing     /usr/bin/chattr
.......T  c /etc/modprobe.d/blacklist-firewire
SM5....T  c /etc/sysconfig/rhn/up2date
S.5....T  c /etc/my.cnf
.......T    /usr/lib/mysql/mysql_config
S.5....T  c /etc/httpd/conf.d/welcome.conf
S.5....T  c /etc/httpd/conf/httpd.conf
.M......    /var/log/httpd
.M......    /var/www/cgi-bin
.......T  c /etc/sysconfig/system-config-users

WGET I modified permissions on myself, and the rest seem to be config files.
0
 
farzanjCommented:
Ok.  Not bad.

Unfortunately, you have to install e2fsprogs.  You have to be able to see the file attributes -- lsattr

You have to be able to change them if they are immutable -- chattr

You need the package.  Which RHEL is it? You can download it from the internet
0
 
farzanjCommented:
I think you should have e2progs rpm already install

From the above output you can see the /usr/bin/chattr has been removed

Do you have lsattr?

which lsattr

If so do

lsattr /bin/f

Also if you notice, your crontab is modified-- MD5 checksum does not match and file size is different

Any time any binary has changed -- RED FLAG.

You should uninstall crontab and reinstall it.

Config files should have a separate verification system kept by the security admin.

Most of other files are config files.  Please carefully check all the other changed files with S or 5 in them and make sure they are not binaries.

rpm -qa | xargs rpm -V |  grep [5S]

As you can see, I would be in a much better position to analyse if the system is configured that way
0
 
HappyCactusCommented:
The server SHOULD be taken down immediately after the compromise.
Every cleaning SHOULD be done by reformatting and reinstalling, because you'll never, never be sure that you have removed ALL the backdoors. More, because it's running a potentially compromised shell, kernel or other things you cannot know of, you'll never be sure that what you are seeing with ps, netstat, ls or whatever is REALLY a good information - rootkit are installed to protect the intruder work.
So, better spend your time with something useful as reformatting and reinstalling your server.
All other activity is useless and potentially VERY DANGEROUS.

Hope that helps.
0
 
jkeegan123Author Commented:
I was able to restore CHATTR from backups, and used this command against the CRONTAB and "f" file to remove entries put there by the individual who infiltrated the system (chattr -i /etc/crontab, chattr -i /bin/f).  After these changes, I could modify out the entries to CRONTAB, as well as delete the file "F" which was definitely part of the payload.  I'm fairly certain that nothing else was compromised.
0
 
HappyCactusCommented:
good luck ;-)
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 8
  • 7
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now