Solved

troubleshooting fedora server lockup

Posted on 2011-03-07
10
389 Views
Last Modified: 2012-05-11
I am starting to work with Fedora servers and I am not very well versed in administration of Linux. I had a server that locked up this morning. I could not ping the server and when I got to the monitor there was just a black screen.

I had to do a hard reboot to get the server up and running.

How can I start to troubleshoot the error. I am not sure what logs I would use to start troubleshooting the issue.
0
Comment
Question by:ryan80
  • 5
  • 4
10 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 35060580
Try looking at

/var/log/messages

/var/log/secure

I am not sure how you got locked.

Did you have a prompt on the console?
0
 
LVL 6

Expert Comment

by:_iskywalker_
ID: 35060690
in /var/log/ there are plenty of logs, you should know what they are, as an admin, so study them!
0
 
LVL 12

Author Comment

by:ryan80
ID: 35061734
thanks for the great response iskywalker. I didnt realize that as an admin I should be familiar with a system and that I would have to review logs.

My inexperience with Linux is why I am asking. Of course I should know what the logs are and what they contain. maybe something a little more constructive like references to where I can find information on logs, or what the basic logs are would be helpfull. I know that browse through a thousand different articles or books and find more information on it, and I will, but I am tyring to troubleshoot an issue now. I posted this question to try and narrow down my search and what I have to reasearch.

@farzanj:
thanks,

There was no prompt at the console. The whole system was unresponsive. I am reviewing those logs now for some hint on what caused the issue.
0
Ransomware: The New Cyber Threat & How to Stop It

This infographic explains ransomware, type of malware that blocks access to your files or your systems and holds them hostage until a ransom is paid. It also examines the different types of ransomware and explains what you can do to thwart this sinister online threat.  

 
LVL 31

Expert Comment

by:farzanj
ID: 35062072
A little more info would make it easy for me as I want to help!
0
 
LVL 12

Author Comment

by:ryan80
ID: 35062309
Thanks,

I am looking through those logs now.

From what I am seeing so far, is that there is nothing from the time that the system stopped responding until the reboot. Here is what I have so far:

Message

Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:19:03 server-in-question mountd[2285]: authenticated mount request from xx.x.x.251:874  or /prod/home/prod (/Production)
Mar  7 04:20:21 server-in-question mountd[2285]: authenticated unmount request from xx.x.x.173:753 for /Dev/home/dev (/Development)
Mar  7 08:47:52 server-in-question kernel: imklog 3.20.2, log source = /proc/kmsg started.
Mar  7 08:47:52 server-in-question rsyslogd: [origin software="rsyslogd" swVersion="3.20.2" x-pid="1885" x-info="http://www.rsyslog.com"] restart

the address at xx.x.x.46 is my monitoring server polling snmp. the last time it showed the server responding is at 4:18. Around 8:47 I powercycled the server.

Here are the logs at that time from secure:

Mar  7 04:19:02 server-in-question sshd[6602]: Connection closed by 127.0.0.1
Mar  7 04:20:02 server-in-question sshd[6613]: Connection closed by 127.0.0.1
Mar  7 08:48:24 server-in-question sshd[2356]: Server listening on 0.0.0.0 port 22.
Mar  7 08:48:24 server-in-question sshd[2356]: Server listening on :: port 22.

0
 
LVL 31

Expert Comment

by:farzanj
ID: 35064293
There are a few things that should be considered.

1.  Fedora is NOT a production brand.  For production use either RedHat (if you can pay) or CentOS (free).  Fedora is a testing distribution to get stable RedHat system.

2.  If you have your server running on run level 5 (GUI), you are asking for trouble.  Linux GUIs are not stable.  Servers should run on run level 3.

3.  Run least number of services.  The services you don't need should not be running on your system.
4.  If possible, enable rsyslog for remote logging.
5.  Try this also
http://linux.about.com/library/cmd/blcmdl1_last.htm
0
 
LVL 12

Author Comment

by:ryan80
ID: 35070823
Thanks for the input.

1. Not that I would know any better, but Fedora was what was being used when I arrived. it is version 9. Not sure why it was picked. Once I know my ass from my elbow with Linux, maybe I can recommend that we move to a different distro.

2. the console does have a graphical screenwhen I get there. I have not tired to log in though. In my limited experience I have always worked through CLI with Linux so I have no problem getting rid of this. is there a way to do this after the fact or is this done in the build of installation?

3. Where can i find the config file that list the services that start?

0
 
LVL 31

Accepted Solution

by:
farzanj earned 500 total points
ID: 35070894
chkconfig --list

Open in new window


You can also do this:

ls /etc/rc.d/rc3.d/S*

Open in new window

0
 
LVL 31

Expert Comment

by:farzanj
ID: 35070963
If the system totally hangs, there is very little some can do, except for reboot.

I don't know you have sar enabled on your system or not.  It could perhaps tell you about the historical state of your system.
0
 
LVL 12

Author Closing Comment

by:ryan80
ID: 35073478
thanks for all the help.
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Guacamole cut and paste issue 3 75
linux redhat 7.2 10 74
VMware Workstation 12 Player 16 75
PHP error function not working on AWS 10 11
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

821 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question