Solved

segmentation fault, high server load

Posted on 2009-04-10
7
756 Views
Last Modified: 2013-12-16
I'm having very high server loads, 8 to 14.4.

In investigating, I find this in the httpd/error_log

[Fri Apr 10 08:28:53 2009] [notice] child pid 4591 exit signal Segmentation fault (11)
[Fri Apr 10 08:38:50 2009] [notice] child pid 20513 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:03 2009] [notice] child pid 20032 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:43 2009] [notice] child pid 6393 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:38 2009] [notice] child pid 20525 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:50 2009] [notice] child pid 6394 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:08 2009] [notice] child pid 6910 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:34 2009] [notice] child pid 20233 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:53 2009] [notice] child pid 21073 exit signal Segmentation fault (11}

Is this likely to be related to the problem of the high server load, and what is the next step I should take?

Thank you,

Chris

(this is an older machine btw, Red Hat Linux release 7.3 (Valhalla)) running ensim
0
Comment
Question by:St_Aug_Beach_Bum
  • 5
7 Comments
 

Author Comment

by:St_Aug_Beach_Bum
Comment Utility
Rebooting brings the server load down, and the segmentation fault goes away after rebooting, but this has been going on for two days now, after rebooting it will eventually start happening again (after several hours usually).  

0
 
LVL 40

Accepted Solution

by:
omarfarid earned 500 total points
Comment Utility
if you run top at the problem time, what do you see the processes cpu and memory utilization? Also, what about swap? you could be out of physical memory and swap space.

Are you running apache ? I have see some comments about too many modules loaded and they were pointing to reduce the number of modules loaded.
0
 

Author Comment

by:St_Aug_Beach_Bum
Comment Utility
Thank you for responsding omarfarid.

top confirms low memory, but I can't tell why it's running low on memory.  Is running out of memory causing the segmentation fault, or is that fault eating up the memory?

The load average has been ok for several hours after I rebooted, it just started climbing back up a few minutes ago.

10:44am  up  1:51,  1 user,  load average: 1.89, 1.90, 1.29
94 processes: 91 sleeping, 3 running, 0 zombie, 0 stopped
CPU states:  1.2% user,  3.0% system,  0.0% nice, 95.8% idle
Mem:   506044K av,  500612K used,    5432K free,       0K shrd,   60984K buff
Swap: 2048276K av,   37456K used, 2010820K free                  302772K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 2839 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4646 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4647 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4648 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4649 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4650 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4651 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 2738 apache     9   0 12852  10M  3188 S     0.0  2.0   0:03 httpd
 2695 apache     9   0 12428 9968  3064 S     0.0  1.9   0:04 httpd
 2698 apache     9   0 10380 7932  2636 S     0.0  1.5   0:04 httpd
 6428 apache     9   0 10260 7916  2708 S     0.0  1.5   0:02 httpd
 2696 apache     9   0 10584 7712  2528 S     0.0  1.5   0:03 httpd
 2694 apache    10   0 10264 7660  2620 S     0.4  1.5   0:03 httpd
 2737 apache     9   0 10248 7632  2772 S     0.0  1.5   0:04 httpd
 2697 apache     9   0  9788 7492  2564 S     0.0  1.4   0:03 httpd
 2743 apache     9   0 10084 7380  2724 S     0.0  1.4   0:03 httpd
 2715 apache     9   0  9820 7372  2648 S     0.0  1.4   0:04 httpd
13041 apache     9   0  8624 5788  3212 S     0.0  1.1   0:00 httpd
13275 apache    10   0  8504 5672  3468 S     0.2  1.1   0:00 httpd
13039 apache     9   0  8136 5236  3044 S     0.0  1.0   0:00 httpd
13277 apache     9   0  8012 5076  3176 S     0.0  1.0   0:00 httpd
12772 root      17   0  3676 3452   332 R     1.6  0.6   0:04 sendmail
 2444 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2446 mysql     13  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2447 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2448 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
13293 apache    11   0  6696 2852  1968 S     0.0  0.5   0:00 httpd
 2905 root       9   0  2448 2448  1036 S     0.0  0.4   0:00 poprelayd
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 

Author Comment

by:St_Aug_Beach_Bum
Comment Utility
I don't see any one pid taking up a lot of memory, but there are several instances of ensim-python running, and I don't remember seeing that before, so I don't know if that's normal. Or what to do about it.
0
 

Author Comment

by:St_Aug_Beach_Bum
Comment Utility
Oh, and I do have apache, but I haven't made any changes to the server software or added any new programs for it to run.
0
 

Author Closing Comment

by:St_Aug_Beach_Bum
Comment Utility
Omarfarid, thank you.  The problem pretty much resolved itself, and why we did not really get a 'solution' here, you seem to be correct that it was memory related. So I will award you the points.  Thank you for your assistance, Chris
0
 
LVL 1

Expert Comment

by:zmagyar
Comment Utility
Actually I got the same problem and started on the memory route. The free memory is low here as well. But free memory means nothing on linux as the free memory is usually used as FS cache and freed up immediately as an application needs it.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

This is the error message I got (CODE) Error caused by incompatible libmp3lame 3.98-2 with ffmpeg I've googled this error message and found out sometimes it attaches this note "can be treated with downgrade libmp3lame to version 3.97 or 3.98" …
Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now