?
Solved

Did my Linux lab box get hacked?

Posted on 2014-04-21
17
Medium Priority
?
1,027 Views
Last Modified: 2014-04-22
I have a CentOS box I use for non-critical stuff in the house like house phones on asterisk,  and to play around/learn linux.  Over the weekend, I noticed my home phone server hitting 100% CPU/MEM used.  when I used the "top" command in SSH, I noticed a user "Nagios" killing my resources with the command "w00t" which made me think someone had got me.  The other server was the same but the command was "tester".
Is my box infected or exploited?, and can I save it?

top - 12:53:59 up 23:58,  1 user,  load average: 2.33, 2.37, 2.52
Tasks: 121 total,   2 running, 119 sleeping,   0 stopped,   0 zombie
Cpu(s): 47.3%us,  0.3%sy, 33.9%ni, 13.6%id,  2.1%wa,  2.7%hi,  0.0%si,  0.0%st
Mem:   3893584k total,  3773736k used,   119848k free,     1744k buffers
Swap:   785400k total,   392732k used,   392668k free,    20068k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25253 nagios    20   0 3475m 3.1g  316 R 198.6 84.6 369:06.17 w00t
    1 root      20   0 19232  368  360 S  0.0  0.0   0:00.59 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:05.53 migration/0

Open in new window

top - 18:35:10 up  4:28,  1 user,  load average: 3.14, 3.15, 3.35
Tasks: 145 total,   1 running, 144 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.7%us,  0.5%sy, 97.8%ni,  0.0%id,  0.0%wa,  0.1%hi,  0.0%si,  0.0%
Mem:   1536088k total,   473144k used,  1062944k free,     8704k buffers
Swap:   524280k total,   115948k used,   408332k free,    70516k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20250 nagios    20   0  470m  10m 1128 S 390.8  0.7 155:58.30 tester
 1465 asterisk  20   0  387m  17m 3148 S  4.0  1.2   1:24.05 httpd

Open in new window

0
Comment
Question by:matthewmalk248
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 5
  • 2
  • +2
17 Comments
 
LVL 5

Expert Comment

by:Pasha Kravtsov
ID: 40013622
Hmm yes it sounds as if you have an malicious user on your box
do 'ps faux' and 'w' then do 'netstat -n' and post the info if you feel comfortable.
0
 

Author Comment

by:matthewmalk248
ID: 40013630
ps faux showed some interesting things...

Looks like I have some friends from Singapore playing with my emotions!

nagios    2030  0.0  0.0  39364    88 ?        Ss   14:07   0:03 /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d

Open in new window

nagios   20250  377  0.6 482180 10544 ?        Ssl  17:52 241:18 /tmp/tester -o stratum+tcp://multi1.wemineall.com:80 -u weedee.1 -p x --alg

Open in new window


What's kinda wierd, is "Users" only shows me, not this ghost user, and "w" shows only
-bash-4.1# w
 18:58:15 up  4:51,  1 user,  load average: 4.01, 4.03, 3.85
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    MY.IP.ADDRESS    17:07    0.00s  0.01s  0.00s w

Open in new window


And netstat-n..
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 127.0.0.1:53683             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53666             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53673             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53665             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53677             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53675             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53672             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53680             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53670             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:5038              127.0.0.1:51946             ESTABLISHED
tcp        0      0 127.0.0.1:53674             127.0.0.1:5038              TIME_WAIT
tcp        0     52 MY.SERVERS.IP:22            MY.REMOTED-IN.IP:50462         ESTABLISHED
tcp        0      0 127.0.0.1:53682             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53676             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53687             127.0.0.1:5038              TIME_WAIT
tcp        1      0 MY.SERVERS.IP:5666          180.210.205.209:50903       CLOSE_WAIT
tcp        0      0 127.0.0.1:53664             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53684             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53685             127.0.0.1:5038              TIME_WAIT
tcp        0      0 MY.SERVERS.IP:59670         37.187.28.68:80             ESTABLISHED
tcp        0      0 127.0.0.1:53686             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53678             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53668             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53671             127.0.0.1:5038              TIME_WAIT
tcp        0      0 ::ffff:127.0.0.1:51946      ::ffff:127.0.0.1:5038       ESTABLISHED
tcp        0      0 ::ffff:MY.SERVERS.IP:80     ::ffff:MY.REMOTED-IN.IP:51963  ESTABLISHED

Open in new window

0
 

Author Comment

by:matthewmalk248
ID: 40013636
looks like port 5666 has been exploited, which I'm guessing is Nagios, so there's a Nagios exploit?
0
Are You Using the Best Web Development Editor?

The worlds of web hosting and web development are constantly evolving. Every year we see design trends change, coding standards adapt and new frameworks/CMS created. With such a quick pace of change it’s easy to get lost trying to keep up.

See if your editor made the list.

 
LVL 5

Accepted Solution

by:
Pasha Kravtsov earned 1800 total points
ID: 40013642
Hmm for the time being disable sshd, edit /etc/passwd and remove that user. Kill all the tasks being run by that user and make sure that there aren't any other weird things being ran.
There are two reasons why the user might not be showing up using the 'w' command.
1. He's not currently logged in
2. There is a rootkit on your box and it has that command hooked to not show him.

Use this C program to detect whether there is a rootkit on your machine
http://www.chokepoint.net/2014/02/detecting-userland-preload-rootkits.html (great blog)

??? -> http://www.nagios.org/
0
 
LVL 26

Expert Comment

by:pony10us
ID: 40013681
There is an exploit related to Nagios and NRPE (port 5666).

http://www.opsview.com/whats-new/blog/security-notice-nrpe-exploit
0
 

Author Comment

by:matthewmalk248
ID: 40013692
Well this doesn't look good then, I ran that C program to check for userland rootkits..
[+] Checking open syscall.
[+] Checking readdir syscall.
[+] Checking fopen syscall.
[!] Preload hooks dectected!
Libc address: 0x175410
Next address: 0x239330
[+] Checking accept syscall.
[+] Checking access syscall.
[+] Checking unlink syscall.

Open in new window

0
 
LVL 5

Expert Comment

by:Pasha Kravtsov
ID: 40013697
damn that's bad.. looks like you have a rootkit installed on your box. Best bet is to reinstall the box and back up your critical files.. When I say backup critical data I mean text files and things like that, nothing that is able to carry a infected code.
0
 

Author Comment

by:matthewmalk248
ID: 40013713
Dang! Is there any way to like use rkhunter logs to clean it?
0
 

Author Comment

by:matthewmalk248
ID: 40013741
Also, odd note, I ran it again, and everything seemed normal?  Ran a few more times to be sure.  This was after I had deleted the Nagios user from etc/passwd.

]# ./preloadcheck
[+] Checking open syscall.
[+] Checking readdir syscall.
[+] Checking fopen syscall.
[+] Checking accept syscall.
[+] Checking access syscall.
[+] Checking unlink syscall.

Open in new window

0
 
LVL 79

Expert Comment

by:arnold
ID: 40013780
Nagios is a network/system monitoring tool. If you did not install it, that would be a concern, if you had set it up. You may have a check deigned for a host that could have gone wrong what is thesecond system where tester was running?

Without looking at the code, it seems teaser is suppose to monitor a web site an alert presumably if the site is down, or using a extended test, if the content on the page  changed
Often, one places a marker such as an HTML comment, if that comment is missing, an alert that the website "corrupted", compromised.....
0
 
LVL 5

Expert Comment

by:Pasha Kravtsov
ID: 40013786
Mathew, that program checks whether a rootkit is actively running on your machine. It makes sense after you killed the malicious user that the rootkit died/isn't running anymore.. a little.. haha well try running ps faux and continue monitoring for issues. If you have anymore feel free to visit my profile and send me an email.
0
 

Author Comment

by:matthewmalk248
ID: 40013873
LoL yea , what a wimpy rootkit if it dies when I ran "skill -STOP -u nagios".  I'm going to tighten up the system and monitor it a lot for the next few weeks.  I changed the SSH port, disabled password auth so only using keys now, and set up iptables.  There's nothing life (or credit) threatening on the box, so I'm kind of curious if I can dismantle this infection piece by piece.  Would be kind of cool not having to reset all this stuff up!
0
 
LVL 79

Expert Comment

by:arnold
ID: 40013899
Make sure to update openssl, openssh.  

Did you previously setup nagio? On your system?
0
 

Author Comment

by:matthewmalk248
ID: 40013942
No, but I noticed that there's a "nagios" user in /etc/passwd even on a fresh install of this Distro of CentOS 6.5, is that normal?
0
 
LVL 5

Expert Comment

by:Pasha Kravtsov
ID: 40014050
Hm sounds like you got something that was vulnerable I'm not sure. Definitly not openssl issues.. lol it's not heartbleed
0
 
LVL 38

Assisted Solution

by:Gerwin Jansen, EE MVE
Gerwin Jansen, EE MVE earned 200 total points
ID: 40014785
tcp://multi1.wemineall.com:80 -> Your CPU is being used to generate some bitcoins for someone
0
 

Author Comment

by:matthewmalk248
ID: 40014810
Thanks Gerwin, that's actually a relief! Probably the least malicious thing I can imagine, other than wasting electricity
0

Featured Post

WordPress Tutorial 3: Plugins, Themes, and Widgets

The three most common changes you will make to your website involve the look (themes), the functionality (plugins), and modular elements (widgets).

In this article we will briefly define each again, and give you directions on how to install them.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The conference as a whole was very interesting, although if one has to make a choice between this one and some others, you may want to check out the others.  This conference is aimed mainly at government agencies.  So it addresses the various compli…
With the rising number of cyber attacks in recent years, keeping your personal data safe has become more important than ever. The tips outlined in this article will help you keep your identitfy safe.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses
Course of the Month11 days, 4 hours left to enroll

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question