Linux Security: Mysterious Server Corruption- How to track it down with log files

Posted on 2013-06-14
Last Modified: 2013-06-20
Dear Experts,
I have had the following mysterious problem that has completely corrupted my godaddy server twice in the last two weeks.

Godaddy's tech support gave us this explanation:

"We reviewed the account as to why we couldn't connect via ssh after a reboot and we discovered that all of your files and directories in your server have been modified to 777 permissions. This basically broke you server entirely and this will require a re-provision of your server".

Do you have any advice about how to stop this happening in the future by doing some of the following:

a) Examining the existing log files
b) Raising the detail level of the log files
c) Using a server Telnet client other than PuTTY.exe
d) Some other strategy

We rent our own dedicated server from godaddy "Dedicated Hosting Economy Linux CentOS - i3" .  We also pay a monthly fee for cPanel on top of that.
Best Wishes,
Question by:PTRUSCOTT
LVL 21

Expert Comment

ID: 39249990
Before moving forward and look at other option. First, check /var/log/secure, it should tell you last log on and other important information.

Expert Comment

ID: 39254040

I think that your server could have a security issue.....

If you cannot connect to the corrupted server anymore i would ask goddady for a complete snapshot of the entire harddisk... Than run it in as a VM to exmine the server for security holes...

What services is running your server? I would first check there for security holes...

Also maybe one of your office computers could be infected so someone could steal the credentials of that server...

All that is possible but hard to verify if you don't have the possibility to check what happend on that corrupted server....

Once you have access to that corrupted server you can verify:

1. Firewall
2. CMD history
3. Log Files
4. root kit hunter
5. You could also send the snapshot to a security company for analysis if it's very important for you
LVL 19

Accepted Solution

jools earned 500 total points
ID: 39254732
If you server has been compromised and who ever did it knows what they are doing you may not see anything in the logs, but you need to be checking the log files in /var/log daily, check out entries in the httpd logs as well as this may give you a clue in the first case.

The server needs to be kept up to date and locked down (DONT USE TELNET!!). There may be vulnerabilities in the software you run on the server so keep up to date with patches and updates on any web apps, if possible disable or review any third party addons as well.

It may even be that someone just did a typo on the server and did a recursive 777 from the wrong directory, in any case you need to make sure permissions are correct and perhaps consider using SELinux for some added (though not complete) protection.

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you use Debian 6 Squeeze and you are tired of looking at the childish graphical GDM login screen that is used by default, here's an easy way to change it. If you've already tried to change it you've probably discovered that none of the old met…
BIND is the most widely used Name Server. A Name Server is the one that translates a site name to it's IP address. There is a new bug in BIND (, affecting all versions of BIND 9 from BIND 9.1.0 (inclusive) thro…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

929 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now