Solved

troubleshooting fedora server lockup

Posted on 2011-03-07
10
383 Views
Last Modified: 2012-05-11
I am starting to work with Fedora servers and I am not very well versed in administration of Linux. I had a server that locked up this morning. I could not ping the server and when I got to the monitor there was just a black screen.

I had to do a hard reboot to get the server up and running.

How can I start to troubleshoot the error. I am not sure what logs I would use to start troubleshooting the issue.
0
Comment
Question by:ryan80
  • 5
  • 4
10 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 35060580
Try looking at

/var/log/messages

/var/log/secure

I am not sure how you got locked.

Did you have a prompt on the console?
0
 
LVL 6

Expert Comment

by:_iskywalker_
ID: 35060690
in /var/log/ there are plenty of logs, you should know what they are, as an admin, so study them!
0
 
LVL 12

Author Comment

by:ryan80
ID: 35061734
thanks for the great response iskywalker. I didnt realize that as an admin I should be familiar with a system and that I would have to review logs.

My inexperience with Linux is why I am asking. Of course I should know what the logs are and what they contain. maybe something a little more constructive like references to where I can find information on logs, or what the basic logs are would be helpfull. I know that browse through a thousand different articles or books and find more information on it, and I will, but I am tyring to troubleshoot an issue now. I posted this question to try and narrow down my search and what I have to reasearch.

@farzanj:
thanks,

There was no prompt at the console. The whole system was unresponsive. I am reviewing those logs now for some hint on what caused the issue.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35062072
A little more info would make it easy for me as I want to help!
0
 
LVL 12

Author Comment

by:ryan80
ID: 35062309
Thanks,

I am looking through those logs now.

From what I am seeing so far, is that there is nothing from the time that the system stopped responding until the reboot. Here is what I have so far:

Message

Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:18:51 server-in-question snmpd[2118]: Connection from UDP: [xx.x.x.46]:2225
Mar  7 04:19:03 server-in-question mountd[2285]: authenticated mount request from xx.x.x.251:874  or /prod/home/prod (/Production)
Mar  7 04:20:21 server-in-question mountd[2285]: authenticated unmount request from xx.x.x.173:753 for /Dev/home/dev (/Development)
Mar  7 08:47:52 server-in-question kernel: imklog 3.20.2, log source = /proc/kmsg started.
Mar  7 08:47:52 server-in-question rsyslogd: [origin software="rsyslogd" swVersion="3.20.2" x-pid="1885" x-info="http://www.rsyslog.com"] restart

the address at xx.x.x.46 is my monitoring server polling snmp. the last time it showed the server responding is at 4:18. Around 8:47 I powercycled the server.

Here are the logs at that time from secure:

Mar  7 04:19:02 server-in-question sshd[6602]: Connection closed by 127.0.0.1
Mar  7 04:20:02 server-in-question sshd[6613]: Connection closed by 127.0.0.1
Mar  7 08:48:24 server-in-question sshd[2356]: Server listening on 0.0.0.0 port 22.
Mar  7 08:48:24 server-in-question sshd[2356]: Server listening on :: port 22.

0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 31

Expert Comment

by:farzanj
ID: 35064293
There are a few things that should be considered.

1.  Fedora is NOT a production brand.  For production use either RedHat (if you can pay) or CentOS (free).  Fedora is a testing distribution to get stable RedHat system.

2.  If you have your server running on run level 5 (GUI), you are asking for trouble.  Linux GUIs are not stable.  Servers should run on run level 3.

3.  Run least number of services.  The services you don't need should not be running on your system.
4.  If possible, enable rsyslog for remote logging.
5.  Try this also
http://linux.about.com/library/cmd/blcmdl1_last.htm
0
 
LVL 12

Author Comment

by:ryan80
ID: 35070823
Thanks for the input.

1. Not that I would know any better, but Fedora was what was being used when I arrived. it is version 9. Not sure why it was picked. Once I know my ass from my elbow with Linux, maybe I can recommend that we move to a different distro.

2. the console does have a graphical screenwhen I get there. I have not tired to log in though. In my limited experience I have always worked through CLI with Linux so I have no problem getting rid of this. is there a way to do this after the fact or is this done in the build of installation?

3. Where can i find the config file that list the services that start?

0
 
LVL 31

Accepted Solution

by:
farzanj earned 500 total points
ID: 35070894
chkconfig --list

Open in new window


You can also do this:

ls /etc/rc.d/rc3.d/S*

Open in new window

0
 
LVL 31

Expert Comment

by:farzanj
ID: 35070963
If the system totally hangs, there is very little some can do, except for reboot.

I don't know you have sar enabled on your system or not.  It could perhaps tell you about the historical state of your system.
0
 
LVL 12

Author Closing Comment

by:ryan80
ID: 35073478
thanks for all the help.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

Over the last ten+ years I have seen Linux configuration tools come and go. In the early days there was the tried-and-true, all-powerful linuxconf that many thought would remain the one and only Linux configuration tool until the end of times. Well,…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now