Did my Linux lab box get hacked?

Hmm yes it sounds as if you have an malicious user on your box
do 'ps faux' and 'w' then do 'netstat -n' and post the info if you feel comfortable.

ps faux showed some interesting things...

Looks like I have some friends from Singapore playing with my emotions!

nagios    2030  0.0  0.0  39364    88 ?        Ss   14:07   0:03 /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d

Select allOpen in new window

nagios   20250  377  0.6 482180 10544 ?        Ssl  17:52 241:18 /tmp/tester -o stratum+tcp://multi1.wemineall.com:80 -u weedee.1 -p x --alg

Select allOpen in new window

What's kinda wierd, is "Users" only shows me, not this ghost user, and "w" shows only

-bash-4.1# w
 18:58:15 up  4:51,  1 user,  load average: 4.01, 4.03, 3.85
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    MY.IP.ADDRESS    17:07    0.00s  0.01s  0.00s w

Select allOpen in new window

And netstat-n..

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 127.0.0.1:53683             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53666             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53673             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53665             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53677             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53675             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53672             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53680             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53670             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:5038              127.0.0.1:51946             ESTABLISHED
tcp        0      0 127.0.0.1:53674             127.0.0.1:5038              TIME_WAIT
tcp        0     52 MY.SERVERS.IP:22            MY.REMOTED-IN.IP:50462         ESTABLISHED
tcp        0      0 127.0.0.1:53682             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53676             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53687             127.0.0.1:5038              TIME_WAIT
tcp        1      0 MY.SERVERS.IP:5666          180.210.205.209:50903       CLOSE_WAIT
tcp        0      0 127.0.0.1:53664             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53684             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53685             127.0.0.1:5038              TIME_WAIT
tcp        0      0 MY.SERVERS.IP:59670         37.187.28.68:80             ESTABLISHED
tcp        0      0 127.0.0.1:53686             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53678             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53668             127.0.0.1:5038              TIME_WAIT
tcp        0      0 127.0.0.1:53671             127.0.0.1:5038              TIME_WAIT
tcp        0      0 ::ffff:127.0.0.1:51946      ::ffff:127.0.0.1:5038       ESTABLISHED
tcp        0      0 ::ffff:MY.SERVERS.IP:80     ::ffff:MY.REMOTED-IN.IP:51963  ESTABLISHED

Select allOpen in new window

looks like port 5666 has been exploited, which I'm guessing is Nagios, so there's a Nagios exploit?

There is an exploit related to Nagios and NRPE (port 5666).

http://www.opsview.com/whats-new/blog/security-notice-nrpe-exploit

Well this doesn't look good then, I ran that C program to check for userland rootkits..

[+] Checking open syscall.
[+] Checking readdir syscall.
[+] Checking fopen syscall.
[!] Preload hooks dectected!
Libc address: 0x175410
Next address: 0x239330
[+] Checking accept syscall.
[+] Checking access syscall.
[+] Checking unlink syscall.

Select allOpen in new window

damn that's bad.. looks like you have a rootkit installed on your box. Best bet is to reinstall the box and back up your critical files.. When I say backup critical data I mean text files and things like that, nothing that is able to carry a infected code.

Dang! Is there any way to like use rkhunter logs to clean it?

Also, odd note, I ran it again, and everything seemed normal?  Ran a few more times to be sure.  This was after I had deleted the Nagios user from etc/passwd.

]# ./preloadcheck
[+] Checking open syscall.
[+] Checking readdir syscall.
[+] Checking fopen syscall.
[+] Checking accept syscall.
[+] Checking access syscall.
[+] Checking unlink syscall.

Select allOpen in new window

Nagios is a network/system monitoring tool. If you did not install it, that would be a concern, if you had set it up. You may have a check deigned for a host that could have gone wrong what is thesecond system where tester was running?

Without looking at the code, it seems teaser is suppose to monitor a web site an alert presumably if the site is down, or using a extended test, if the content on the page  changed
Often, one places a marker such as an HTML comment, if that comment is missing, an alert that the website "corrupted", compromised.....

Mathew, that program checks whether a rootkit is actively running on your machine. It makes sense after you killed the malicious user that the rootkit died/isn't running anymore.. a little.. haha well try running ps faux and continue monitoring for issues. If you have anymore feel free to visit my profile and send me an email.

LoL yea , what a wimpy rootkit if it dies when I ran "skill -STOP -u nagios".  I'm going to tighten up the system and monitor it a lot for the next few weeks.  I changed the SSH port, disabled password auth so only using keys now, and set up iptables.  There's nothing life (or credit) threatening on the box, so I'm kind of curious if I can dismantle this infection piece by piece.  Would be kind of cool not having to reset all this stuff up!

Make sure to update openssl, openssh.  

Did you previously setup nagio? On your system?

No, but I noticed that there's a "nagios" user in /etc/passwd even on a fresh install of this Distro of CentOS 6.5, is that normal?

Hm sounds like you got something that was vulnerable I'm not sure. Definitly not openssl issues.. lol it's not heartbleed

Thanks Gerwin, that's actually a relief! Probably the least malicious thing I can imagine, other than wasting electricity