We help IT Professionals succeed at work.

Linux Server Compromised - now I can't delete or rename (2) files

Last Modified: 2012-05-11
I recently had a client have their Linux box compromised due to a globally open SSH port and a weak root password.

After the compromise, a CRON job was left running every minute that started a process that logged the server into an IRC channel in order to become part of a BOTNET or some other such situation.  Regardless, I was able to kill all processes related to this by killing the CRON process and all of the processes that were spawned by this.

I have (2) things left to clean up: the CRONTAB file still holds entries in it for starting a specific process named "/bin/f" , and I am UNABLE to modify the CRONTAB file (access is denied) and I am UNABLE to delete / rename the file "/bin/f".  The files permissions are as follows:

-rw-r--r--   1 root root     283 Apr 17 01:59 crontab

-rwxr-xr-x  1 root root  885401 Feb 18 01:19 f

I have tried to CHMOD the files, and I get "changing permissions of "f": Operation not permitted.
I have tried to CHOWN the files, and I get the same thing:  Operation not permitted.

Can someone tell me what I have to do to get rid of these files?  I have looked at processes running with ps -aux and I don't see anything running "/bin/f" anymore, and there are no crontab processes running, so I don't think that "/etc/crontab" is being held open.  

All of the volumes are ext3 file systems, here is the output of mount:

/dev/sda5 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda3 on /var/log type ext3 (rw)
/dev/sda2 on /var/lib/mysql type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

Watch Question

You use sudo and still says access denied ?

try to see if there is any new user has been created.


I'm in as root, and cannot make these changes to these files.  I can make changes to ALL other files still.

Yes, I also tried performing these operations using SUDO and the same thing happens.  

It is not my box so I'm not sure if there are new users, but it didn't look like anything new was there.  All of this was done with root and the weak password that it had.


From tags it appears that it is Red Hat linux.

You have to verify RPMs.

Often in attacks, the binaries are changed.  So the ls is no more a ls -- it would not show some of files the attacher wants to keep.  ps may not be ps as it may not show some of the processes the attacker doesn't want you to see.

In your case rm doesn't appear to be the actual binary.

If the system was not very properly protected, your best approach would be to reinstall the OS.

Short of that, you can start verifying RPMS.  RPM is a very good mechanism as you can verify it and it can tell you what changes were made after a certain time or after the original install.   RPMS are also signed so if anything changes, you can check the signatures.  Any package that doesn't pass those tests should be removed.


try this first

sudo chattr -i /bin/f

and then

sudo rm /bin/f


I was able to use "kill" to end the cron job that was spawning these processes, and other operations seem to work as expected.  I think this was just a lucky hit from a script kid.  Any ideas on those files?


chattr does not appear to be a valid command on this install.

I repeate that you should check if these files are immutable.

chattr +i your_file makes the file immutable and even root user cannot delete them

to reverse that you have to run chattr -i your_file

You should issue this command as noted above:

chattr -i <FILE>

rm -f <FILE>

The problem is that the person also removed a major rpm as well.

Please install this:
yum install e2fsprogs


@farzanj:  Will this disrupt any of the running OS (yum install e2fsprogs)?  Would it be easier to just find a copy of "chattr" to use rather than download the entire "e2fsprogs" ?

It should not disrupt anything.  It has some very basic programs.  In Red Hat, you should always install packages, not files.

You can verify your packages with rpm verification.

rpm -qa | xargs rpm -V

For more information:
man rpm


@farzanj:  When I ran "yum  install e2fsprogs" I got the following error:

Error Message:
        Please run rhn_register as root on this client
Error Class Code: 9
Error Class Info: Invalid System Credentials.
     An error has occurred while processing your request. If this problem
     persists please enter a bug report at bugzilla.redhat.com.
     If you choose to submit the bug report, please be sure to include
     details of what you were trying to do when this error occurred and
     details on how to reproduce this problem.

Setting up Install Process
Parsing package install arguments
Package e2fsprogs-1.39-20.el5.i386 installed and not available
Nothing to do

The above would tell you the changes that were made to your rpm packages.  You should dump it into a file and carefully review it.  You can ignore reviewing documentation changes with
rpm -qa | xargs rpm -V --excludedocs


Was your system registered to RHN before the attack.  It may be a little more than just a lucky attack.  The message that you received is a red flag.  Sorry!


I am CERTAIN that this particular client was not registered to RHN ever.  You know the type I'm sure.

The results of rpm -qa | xargs rpm -V are:

.M......    /usr/bin/wget
S.5....T  c /etc/crontab
.......T  c /etc/mail/sendmail.cf
S.5....T  c /var/log/mail/statistics
......G.    /var/cache/samba/winbindd_privileged
S.5....T  c /etc/ppp/chap-secrets
S.5....T  c /etc/ppp/pap-secrets
.......T  c /etc/audit/auditd.conf
S.5....T    /usr/share/logwatch/default.conf/logwatch.conf
S.5....T    /usr/lib/qt-3.3/etc/settings/qtrc
S.5....T    /usr/share/icons/hicolor/icon-theme.cache
S.5....T  c /etc/sudoers
S.5....T  c /etc/freetds.conf
S.5....T  c /etc/printcap
SM5....T  c /etc/sysconfig/iptables-config
....L...  c /etc/pam.d/system-auth
..5....T  c /etc/inittab
S.5....T  c /etc/xml/catalog
S.5....T  c /usr/share/sgml/docbook/xmlcatalog
S.5....T  c /etc/php.ini
S.5....T  c /etc/sysconfig/system-config-securitylevel
Unsatisfied dependencies for libtiff-devel-3.8.2-7.el5.i386: libtiff = 3.8.2-7.el5
..5....T  c /usr/lib/security/classpath.security
S.5....T  c /etc/sane.d/dll.conf
S.5....T  c /etc/sysconfig/vncservers
.......T    /usr/share/pear/.depdb
.......T    /usr/share/pear/.depdblock
..5....T    /usr/share/pear/.filemap
.......T    /usr/share/pear/.lock
missing     /usr/bin/chattr
.......T  c /etc/modprobe.d/blacklist-firewire
SM5....T  c /etc/sysconfig/rhn/up2date
S.5....T  c /etc/my.cnf
.......T    /usr/lib/mysql/mysql_config
S.5....T  c /etc/httpd/conf.d/welcome.conf
S.5....T  c /etc/httpd/conf/httpd.conf
.M......    /var/log/httpd
.M......    /var/www/cgi-bin
.......T  c /etc/sysconfig/system-config-users

WGET I modified permissions on myself, and the rest seem to be config files.

Ok.  Not bad.

Unfortunately, you have to install e2fsprogs.  You have to be able to see the file attributes -- lsattr

You have to be able to change them if they are immutable -- chattr

You need the package.  Which RHEL is it? You can download it from the internet
Unlock this solution and get a sample of our free trial.
(No credit card required)
The server SHOULD be taken down immediately after the compromise.
Every cleaning SHOULD be done by reformatting and reinstalling, because you'll never, never be sure that you have removed ALL the backdoors. More, because it's running a potentially compromised shell, kernel or other things you cannot know of, you'll never be sure that what you are seeing with ps, netstat, ls or whatever is REALLY a good information - rootkit are installed to protect the intruder work.
So, better spend your time with something useful as reformatting and reinstalling your server.
All other activity is useless and potentially VERY DANGEROUS.

Hope that helps.


I was able to restore CHATTR from backups, and used this command against the CRONTAB and "f" file to remove entries put there by the individual who infiltrated the system (chattr -i /etc/crontab, chattr -i /bin/f).  After these changes, I could modify out the entries to CRONTAB, as well as delete the file "F" which was definitely part of the payload.  I'm fairly certain that nothing else was compromised.
good luck ;-)
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.