Solved

segmentation fault, high server load

Posted on 2009-04-10
7
763 Views
Last Modified: 2013-12-16
I'm having very high server loads, 8 to 14.4.

In investigating, I find this in the httpd/error_log

[Fri Apr 10 08:28:53 2009] [notice] child pid 4591 exit signal Segmentation fault (11)
[Fri Apr 10 08:38:50 2009] [notice] child pid 20513 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:03 2009] [notice] child pid 20032 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:43 2009] [notice] child pid 6393 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:38 2009] [notice] child pid 20525 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:50 2009] [notice] child pid 6394 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:08 2009] [notice] child pid 6910 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:34 2009] [notice] child pid 20233 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:53 2009] [notice] child pid 21073 exit signal Segmentation fault (11}

Is this likely to be related to the problem of the high server load, and what is the next step I should take?

Thank you,

Chris

(this is an older machine btw, Red Hat Linux release 7.3 (Valhalla)) running ensim
0
Comment
Question by:St_Aug_Beach_Bum
  • 5
7 Comments
 

Author Comment

by:St_Aug_Beach_Bum
ID: 24115645
Rebooting brings the server load down, and the segmentation fault goes away after rebooting, but this has been going on for two days now, after rebooting it will eventually start happening again (after several hours usually).  

0
 
LVL 40

Accepted Solution

by:
omarfarid earned 500 total points
ID: 24115773
if you run top at the problem time, what do you see the processes cpu and memory utilization? Also, what about swap? you could be out of physical memory and swap space.

Are you running apache ? I have see some comments about too many modules loaded and they were pointing to reduce the number of modules loaded.
0
 

Author Comment

by:St_Aug_Beach_Bum
ID: 24116485
Thank you for responsding omarfarid.

top confirms low memory, but I can't tell why it's running low on memory.  Is running out of memory causing the segmentation fault, or is that fault eating up the memory?

The load average has been ok for several hours after I rebooted, it just started climbing back up a few minutes ago.

10:44am  up  1:51,  1 user,  load average: 1.89, 1.90, 1.29
94 processes: 91 sleeping, 3 running, 0 zombie, 0 stopped
CPU states:  1.2% user,  3.0% system,  0.0% nice, 95.8% idle
Mem:   506044K av,  500612K used,    5432K free,       0K shrd,   60984K buff
Swap: 2048276K av,   37456K used, 2010820K free                  302772K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 2839 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4646 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4647 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4648 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4649 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4650 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4651 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 2738 apache     9   0 12852  10M  3188 S     0.0  2.0   0:03 httpd
 2695 apache     9   0 12428 9968  3064 S     0.0  1.9   0:04 httpd
 2698 apache     9   0 10380 7932  2636 S     0.0  1.5   0:04 httpd
 6428 apache     9   0 10260 7916  2708 S     0.0  1.5   0:02 httpd
 2696 apache     9   0 10584 7712  2528 S     0.0  1.5   0:03 httpd
 2694 apache    10   0 10264 7660  2620 S     0.4  1.5   0:03 httpd
 2737 apache     9   0 10248 7632  2772 S     0.0  1.5   0:04 httpd
 2697 apache     9   0  9788 7492  2564 S     0.0  1.4   0:03 httpd
 2743 apache     9   0 10084 7380  2724 S     0.0  1.4   0:03 httpd
 2715 apache     9   0  9820 7372  2648 S     0.0  1.4   0:04 httpd
13041 apache     9   0  8624 5788  3212 S     0.0  1.1   0:00 httpd
13275 apache    10   0  8504 5672  3468 S     0.2  1.1   0:00 httpd
13039 apache     9   0  8136 5236  3044 S     0.0  1.0   0:00 httpd
13277 apache     9   0  8012 5076  3176 S     0.0  1.0   0:00 httpd
12772 root      17   0  3676 3452   332 R     1.6  0.6   0:04 sendmail
 2444 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2446 mysql     13  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2447 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2448 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
13293 apache    11   0  6696 2852  1968 S     0.0  0.5   0:00 httpd
 2905 root       9   0  2448 2448  1036 S     0.0  0.4   0:00 poprelayd
0
Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.

 

Author Comment

by:St_Aug_Beach_Bum
ID: 24116506
I don't see any one pid taking up a lot of memory, but there are several instances of ensim-python running, and I don't remember seeing that before, so I don't know if that's normal. Or what to do about it.
0
 

Author Comment

by:St_Aug_Beach_Bum
ID: 24116560
Oh, and I do have apache, but I haven't made any changes to the server software or added any new programs for it to run.
0
 

Author Closing Comment

by:St_Aug_Beach_Bum
ID: 31568896
Omarfarid, thank you.  The problem pretty much resolved itself, and why we did not really get a 'solution' here, you seem to be correct that it was memory related. So I will award you the points.  Thank you for your assistance, Chris
0
 
LVL 1

Expert Comment

by:zmagyar
ID: 24704849
Actually I got the same problem and started on the memory route. The free memory is low here as well. But free memory means nothing on linux as the free memory is usually used as FS cache and freed up immediately as an application needs it.
0

Featured Post

The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
AWS RDS 3 83
Check if a filw is immutable for a certain amount of time 6 61
High Bandwidth Usage 6 53
Can't ping New Linux Servers 40 65
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

808 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question