troubleshooting fedora server lockup

I am starting to work with Fedora servers and I am not very well versed in administration of Linux. I had a server that locked up this morning. I could not ping the server and when I got to the monitor there was just a black screen.

I had to do a hard reboot to get the server up and running.

How can I start to troubleshoot the error. I am not sure what logs I would use to start troubleshooting the issue.
LVL 12
ryan80Asked:
Who is Participating?

[Webinar] Streamline your web hosting managementRegister Today

x
 
farzanjConnect With a Mentor Commented:
chkconfig --list

Open in new window


You can also do this:

ls /etc/rc.d/rc3.d/S*

Open in new window

0
 
farzanjCommented:
Try looking at

/var/log/messages

/var/log/secure

I am not sure how you got locked.

Did you have a prompt on the console?
0
 
_iskywalker_Commented:
in /var/log/ there are plenty of logs, you should know what they are, as an admin, so study them!
0
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

 
ryan80Author Commented:
thanks for the great response iskywalker. I didnt realize that as an admin I should be familiar with a system and that I would have to review logs.

My inexperience with Linux is why I am asking. Of course I should know what the logs are and what they contain. maybe something a little more constructive like references to where I can find information on logs, or what the basic logs are would be helpfull. I know that browse through a thousand different articles or books and find more information on it, and I will, but I am tyring to troubleshoot an issue now. I posted this question to try and narrow down my search and what I have to reasearch.

@farzanj:
thanks,

There was no prompt at the console. The whole system was unresponsive. I am reviewing those logs now for some hint on what caused the issue.
0
 
farzanjCommented:
A little more info would make it easy for me as I want to help!
0
 
ryan80Author Commented:
Thanks,

I am looking through those logs now.

From what I am seeing so far, is that there is nothing from the time that the system stopped responding until the reboot. Here is what I have so far:

Message

Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:19:03 server-in-question mountd[2285]: authenticated mount request from xx.x.x.251:874  or /prod/home/prod (/Production)
Mar  7 04:20:21 server-in-question mountd[2285]: authenticated unmount request from xx.x.x.173:753 for /Dev/home/dev (/Development)
Mar  7 08:47:52 server-in-question kernel: imklog 3.20.2, log source = /proc/kmsg started.
Mar  7 08:47:52 server-in-question rsyslogd: [origin software="rsyslogd" swVersion="3.20.2" x-pid="1885" x-info="http://www.rsyslog.com"] restart

the address at xx.x.x.46 is my monitoring server polling snmp. the last time it showed the server responding is at 4:18. Around 8:47 I powercycled the server.

Here are the logs at that time from secure:

Mar  7 04:19:02 server-in-question sshd[6602]: Connection closed by 127.0.0.1
Mar  7 04:20:02 server-in-question sshd[6613]: Connection closed by 127.0.0.1
Mar  7 08:48:24 server-in-question sshd[2356]: Server listening on 0.0.0.0 port 22.
Mar  7 08:48:24 server-in-question sshd[2356]: Server listening on :: port 22.

0
 
farzanjCommented:
There are a few things that should be considered.

1.  Fedora is NOT a production brand.  For production use either RedHat (if you can pay) or CentOS (free).  Fedora is a testing distribution to get stable RedHat system.

2.  If you have your server running on run level 5 (GUI), you are asking for trouble.  Linux GUIs are not stable.  Servers should run on run level 3.

3.  Run least number of services.  The services you don't need should not be running on your system.
4.  If possible, enable rsyslog for remote logging.
5.  Try this also
http://linux.about.com/library/cmd/blcmdl1_last.htm
0
 
ryan80Author Commented:
Thanks for the input.

1. Not that I would know any better, but Fedora was what was being used when I arrived. it is version 9. Not sure why it was picked. Once I know my ass from my elbow with Linux, maybe I can recommend that we move to a different distro.

2. the console does have a graphical screenwhen I get there. I have not tired to log in though. In my limited experience I have always worked through CLI with Linux so I have no problem getting rid of this. is there a way to do this after the fact or is this done in the build of installation?

3. Where can i find the config file that list the services that start?

0
 
farzanjCommented:
If the system totally hangs, there is very little some can do, except for reboot.

I don't know you have sar enabled on your system or not.  It could perhaps tell you about the historical state of your system.
0
 
ryan80Author Commented:
thanks for all the help.
0
All Courses

From novice to tech pro — start learning today.