Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

segmentation fault, high server load

Posted on 2009-04-10
7
Medium Priority
?
781 Views
Last Modified: 2013-12-16
I'm having very high server loads, 8 to 14.4.

In investigating, I find this in the httpd/error_log

[Fri Apr 10 08:28:53 2009] [notice] child pid 4591 exit signal Segmentation fault (11)
[Fri Apr 10 08:38:50 2009] [notice] child pid 20513 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:03 2009] [notice] child pid 20032 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:43 2009] [notice] child pid 6393 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:38 2009] [notice] child pid 20525 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:50 2009] [notice] child pid 6394 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:08 2009] [notice] child pid 6910 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:34 2009] [notice] child pid 20233 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:53 2009] [notice] child pid 21073 exit signal Segmentation fault (11}

Is this likely to be related to the problem of the high server load, and what is the next step I should take?

Thank you,

Chris

(this is an older machine btw, Red Hat Linux release 7.3 (Valhalla)) running ensim
0
Comment
Question by:St_Aug_Beach_Bum
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
7 Comments
 

Author Comment

by:St_Aug_Beach_Bum
ID: 24115645
Rebooting brings the server load down, and the segmentation fault goes away after rebooting, but this has been going on for two days now, after rebooting it will eventually start happening again (after several hours usually).  

0
 
LVL 40

Accepted Solution

by:
omarfarid earned 2000 total points
ID: 24115773
if you run top at the problem time, what do you see the processes cpu and memory utilization? Also, what about swap? you could be out of physical memory and swap space.

Are you running apache ? I have see some comments about too many modules loaded and they were pointing to reduce the number of modules loaded.
0
 

Author Comment

by:St_Aug_Beach_Bum
ID: 24116485
Thank you for responsding omarfarid.

top confirms low memory, but I can't tell why it's running low on memory.  Is running out of memory causing the segmentation fault, or is that fault eating up the memory?

The load average has been ok for several hours after I rebooted, it just started climbing back up a few minutes ago.

10:44am  up  1:51,  1 user,  load average: 1.89, 1.90, 1.29
94 processes: 91 sleeping, 3 running, 0 zombie, 0 stopped
CPU states:  1.2% user,  3.0% system,  0.0% nice, 95.8% idle
Mem:   506044K av,  500612K used,    5432K free,       0K shrd,   60984K buff
Swap: 2048276K av,   37456K used, 2010820K free                  302772K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 2839 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4646 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4647 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4648 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4649 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4650 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 4651 root       5 -10 21372  11M  1208 S <   0.0  2.3   0:00 ensim-python
 2738 apache     9   0 12852  10M  3188 S     0.0  2.0   0:03 httpd
 2695 apache     9   0 12428 9968  3064 S     0.0  1.9   0:04 httpd
 2698 apache     9   0 10380 7932  2636 S     0.0  1.5   0:04 httpd
 6428 apache     9   0 10260 7916  2708 S     0.0  1.5   0:02 httpd
 2696 apache     9   0 10584 7712  2528 S     0.0  1.5   0:03 httpd
 2694 apache    10   0 10264 7660  2620 S     0.4  1.5   0:03 httpd
 2737 apache     9   0 10248 7632  2772 S     0.0  1.5   0:04 httpd
 2697 apache     9   0  9788 7492  2564 S     0.0  1.4   0:03 httpd
 2743 apache     9   0 10084 7380  2724 S     0.0  1.4   0:03 httpd
 2715 apache     9   0  9820 7372  2648 S     0.0  1.4   0:04 httpd
13041 apache     9   0  8624 5788  3212 S     0.0  1.1   0:00 httpd
13275 apache    10   0  8504 5672  3468 S     0.2  1.1   0:00 httpd
13039 apache     9   0  8136 5236  3044 S     0.0  1.0   0:00 httpd
13277 apache     9   0  8012 5076  3176 S     0.0  1.0   0:00 httpd
12772 root      17   0  3676 3452   332 R     1.6  0.6   0:04 sendmail
 2444 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2446 mysql     13  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2447 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
 2448 mysql     15  10  5336 2924  1520 S N   0.0  0.5   0:00 mysqld
13293 apache    11   0  6696 2852  1968 S     0.0  0.5   0:00 httpd
 2905 root       9   0  2448 2448  1036 S     0.0  0.4   0:00 poprelayd
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:St_Aug_Beach_Bum
ID: 24116506
I don't see any one pid taking up a lot of memory, but there are several instances of ensim-python running, and I don't remember seeing that before, so I don't know if that's normal. Or what to do about it.
0
 

Author Comment

by:St_Aug_Beach_Bum
ID: 24116560
Oh, and I do have apache, but I haven't made any changes to the server software or added any new programs for it to run.
0
 

Author Closing Comment

by:St_Aug_Beach_Bum
ID: 31568896
Omarfarid, thank you.  The problem pretty much resolved itself, and why we did not really get a 'solution' here, you seem to be correct that it was memory related. So I will award you the points.  Thank you for your assistance, Chris
0
 
LVL 1

Expert Comment

by:zmagyar
ID: 24704849
Actually I got the same problem and started on the memory route. The free memory is low here as well. But free memory means nothing on linux as the free memory is usually used as FS cache and freed up immediately as an application needs it.
0

Featured Post

Fill in the form and get your FREE NFR key NOW!

Veeam® is happy to provide a FREE NFR server license to certified engineers, trainers, and bloggers.  It allows for the non‑production use of Veeam Agent for Microsoft Windows. This license is valid for five workstations and two servers.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

rdate is a Linux command and the network time protocol for immediate date and time setup from another machine. The clocks are synchronized by entering rdate with the -s switch (command without switch just checks the time but does not set anything). …
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question