Linux Security: Mysterious Server Corruption- How to track it down with log files

Posted on 2013-06-14
Last Modified: 2013-06-20
Dear Experts,
I have had the following mysterious problem that has completely corrupted my godaddy server twice in the last two weeks.

Godaddy's tech support gave us this explanation:

"We reviewed the account as to why we couldn't connect via ssh after a reboot and we discovered that all of your files and directories in your server have been modified to 777 permissions. This basically broke you server entirely and this will require a re-provision of your server".

Do you have any advice about how to stop this happening in the future by doing some of the following:

a) Examining the existing log files
b) Raising the detail level of the log files
c) Using a server Telnet client other than PuTTY.exe
d) Some other strategy

We rent our own dedicated server from godaddy "Dedicated Hosting Economy Linux CentOS - i3" .  We also pay a monthly fee for cPanel on top of that.
Best Wishes,
Question by:PTRUSCOTT
LVL 21

Expert Comment

ID: 39249990
Before moving forward and look at other option. First, check /var/log/secure, it should tell you last log on and other important information.

Expert Comment

ID: 39254040

I think that your server could have a security issue.....

If you cannot connect to the corrupted server anymore i would ask goddady for a complete snapshot of the entire harddisk... Than run it in as a VM to exmine the server for security holes...

What services is running your server? I would first check there for security holes...

Also maybe one of your office computers could be infected so someone could steal the credentials of that server...

All that is possible but hard to verify if you don't have the possibility to check what happend on that corrupted server....

Once you have access to that corrupted server you can verify:

1. Firewall
2. CMD history
3. Log Files
4. root kit hunter
5. You could also send the snapshot to a security company for analysis if it's very important for you
LVL 19

Accepted Solution

jools earned 500 total points
ID: 39254732
If you server has been compromised and who ever did it knows what they are doing you may not see anything in the logs, but you need to be checking the log files in /var/log daily, check out entries in the httpd logs as well as this may give you a clue in the first case.

The server needs to be kept up to date and locked down (DONT USE TELNET!!). There may be vulnerabilities in the software you run on the server so keep up to date with patches and updates on any web apps, if possible disable or review any third party addons as well.

It may even be that someone just did a typo on the server and did a recursive 777 from the wrong directory, in any case you need to make sure permissions are correct and perhaps consider using SELinux for some added (though not complete) protection.

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
maybe no no httpd.conf 6 62
Using sort and uniq to pare down large syslog 6 51
CLI command keep running after close 7 36
Apache module 5 47
In my business, I use the LTS (Long Term Support) versions of Linux. My workstations do real work, and so I rarely have the patience to deal with silly problems caused by an upgraded kernel that had experimental software on it to begin with from a r…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

860 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question