segmentation fault, high server load
I'm having very high server loads, 8 to 14.4.
In investigating, I find this in the httpd/error_log
[Fri Apr 10 08:28:53 2009] [notice] child pid 4591 exit signal Segmentation fault (11)
[Fri Apr 10 08:38:50 2009] [notice] child pid 20513 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:03 2009] [notice] child pid 20032 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:43 2009] [notice] child pid 6393 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:38 2009] [notice] child pid 20525 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:50 2009] [notice] child pid 6394 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:08 2009] [notice] child pid 6910 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:34 2009] [notice] child pid 20233 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:53 2009] [notice] child pid 21073 exit signal Segmentation fault (11}
Is this likely to be related to the problem of the high server load, and what is the next step I should take?
Thank you,
Chris
(this is an older machine btw, Red Hat Linux release 7.3 (Valhalla)) running ensim
In investigating, I find this in the httpd/error_log
[Fri Apr 10 08:28:53 2009] [notice] child pid 4591 exit signal Segmentation fault (11)
[Fri Apr 10 08:38:50 2009] [notice] child pid 20513 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:03 2009] [notice] child pid 20032 exit signal Segmentation fault (11)
[Fri Apr 10 08:39:43 2009] [notice] child pid 6393 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:38 2009] [notice] child pid 20525 exit signal Segmentation fault (11)
[Fri Apr 10 08:40:50 2009] [notice] child pid 6394 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:08 2009] [notice] child pid 6910 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:34 2009] [notice] child pid 20233 exit signal Segmentation fault (11)
[Fri Apr 10 08:41:53 2009] [notice] child pid 21073 exit signal Segmentation fault (11}
Is this likely to be related to the problem of the high server load, and what is the next step I should take?
Thank you,
Chris
(this is an older machine btw, Red Hat Linux release 7.3 (Valhalla)) running ensim
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thank you for responsding omarfarid.
top confirms low memory, but I can't tell why it's running low on memory. Is running out of memory causing the segmentation fault, or is that fault eating up the memory?
The load average has been ok for several hours after I rebooted, it just started climbing back up a few minutes ago.
10:44am up 1:51, 1 user, load average: 1.89, 1.90, 1.29
94 processes: 91 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 1.2% user, 3.0% system, 0.0% nice, 95.8% idle
Mem: 506044K av, 500612K used, 5432K free, 0K shrd, 60984K buff
Swap: 2048276K av, 37456K used, 2010820K free 302772K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
2839 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4646 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4647 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4648 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4649 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4650 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4651 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
2738 apache 9 0 12852 10M 3188 S 0.0 2.0 0:03 httpd
2695 apache 9 0 12428 9968 3064 S 0.0 1.9 0:04 httpd
2698 apache 9 0 10380 7932 2636 S 0.0 1.5 0:04 httpd
6428 apache 9 0 10260 7916 2708 S 0.0 1.5 0:02 httpd
2696 apache 9 0 10584 7712 2528 S 0.0 1.5 0:03 httpd
2694 apache 10 0 10264 7660 2620 S 0.4 1.5 0:03 httpd
2737 apache 9 0 10248 7632 2772 S 0.0 1.5 0:04 httpd
2697 apache 9 0 9788 7492 2564 S 0.0 1.4 0:03 httpd
2743 apache 9 0 10084 7380 2724 S 0.0 1.4 0:03 httpd
2715 apache 9 0 9820 7372 2648 S 0.0 1.4 0:04 httpd
13041 apache 9 0 8624 5788 3212 S 0.0 1.1 0:00 httpd
13275 apache 10 0 8504 5672 3468 S 0.2 1.1 0:00 httpd
13039 apache 9 0 8136 5236 3044 S 0.0 1.0 0:00 httpd
13277 apache 9 0 8012 5076 3176 S 0.0 1.0 0:00 httpd
12772 root 17 0 3676 3452 332 R 1.6 0.6 0:04 sendmail
2444 mysql 15 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
2446 mysql 13 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
2447 mysql 15 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
2448 mysql 15 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
13293 apache 11 0 6696 2852 1968 S 0.0 0.5 0:00 httpd
2905 root 9 0 2448 2448 1036 S 0.0 0.4 0:00 poprelayd
top confirms low memory, but I can't tell why it's running low on memory. Is running out of memory causing the segmentation fault, or is that fault eating up the memory?
The load average has been ok for several hours after I rebooted, it just started climbing back up a few minutes ago.
10:44am up 1:51, 1 user, load average: 1.89, 1.90, 1.29
94 processes: 91 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 1.2% user, 3.0% system, 0.0% nice, 95.8% idle
Mem: 506044K av, 500612K used, 5432K free, 0K shrd, 60984K buff
Swap: 2048276K av, 37456K used, 2010820K free 302772K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
2839 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4646 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4647 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4648 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4649 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4650 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
4651 root 5 -10 21372 11M 1208 S < 0.0 2.3 0:00 ensim-python
2738 apache 9 0 12852 10M 3188 S 0.0 2.0 0:03 httpd
2695 apache 9 0 12428 9968 3064 S 0.0 1.9 0:04 httpd
2698 apache 9 0 10380 7932 2636 S 0.0 1.5 0:04 httpd
6428 apache 9 0 10260 7916 2708 S 0.0 1.5 0:02 httpd
2696 apache 9 0 10584 7712 2528 S 0.0 1.5 0:03 httpd
2694 apache 10 0 10264 7660 2620 S 0.4 1.5 0:03 httpd
2737 apache 9 0 10248 7632 2772 S 0.0 1.5 0:04 httpd
2697 apache 9 0 9788 7492 2564 S 0.0 1.4 0:03 httpd
2743 apache 9 0 10084 7380 2724 S 0.0 1.4 0:03 httpd
2715 apache 9 0 9820 7372 2648 S 0.0 1.4 0:04 httpd
13041 apache 9 0 8624 5788 3212 S 0.0 1.1 0:00 httpd
13275 apache 10 0 8504 5672 3468 S 0.2 1.1 0:00 httpd
13039 apache 9 0 8136 5236 3044 S 0.0 1.0 0:00 httpd
13277 apache 9 0 8012 5076 3176 S 0.0 1.0 0:00 httpd
12772 root 17 0 3676 3452 332 R 1.6 0.6 0:04 sendmail
2444 mysql 15 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
2446 mysql 13 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
2447 mysql 15 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
2448 mysql 15 10 5336 2924 1520 S N 0.0 0.5 0:00 mysqld
13293 apache 11 0 6696 2852 1968 S 0.0 0.5 0:00 httpd
2905 root 9 0 2448 2448 1036 S 0.0 0.4 0:00 poprelayd
ASKER
I don't see any one pid taking up a lot of memory, but there are several instances of ensim-python running, and I don't remember seeing that before, so I don't know if that's normal. Or what to do about it.
ASKER
Oh, and I do have apache, but I haven't made any changes to the server software or added any new programs for it to run.
ASKER
Omarfarid, thank you. The problem pretty much resolved itself, and why we did not really get a 'solution' here, you seem to be correct that it was memory related. So I will award you the points. Thank you for your assistance, Chris
Actually I got the same problem and started on the memory route. The free memory is low here as well. But free memory means nothing on linux as the free memory is usually used as FS cache and freed up immediately as an application needs it.
ASKER