Avatar of davidwang69
davidwang69
 asked on

cause of apache runaway process

We are using apache 2.0 as our web server running in SunOS  5.10.

From time to time we have runaway process which will comsume huge amount of CPU, something like 30 - 60%.

If we don't restart the apache instance there will be no access to the web page.

How can we find out what cause the runaway process?
Apache Web Server

Avatar of undefined
Last Comment
davidwang69

8/22/2022 - Mon
gheist

Do you use infinite loop in PHP? mod_rewrite? htaccess?

Are you able to attach debugger to the process to see what is inside?
davidwang69

ASKER
We have 2 web servers and running Vignette as content management system on 2 app servers. We don't have PHP and htaccess but we do have a lot mod_rewrite rule defined.

What is debugger, core dump?
gheist

You mention commercial product - it is a good idea to ask THEIR already paid support to help out.

https://secure.wikimedia.org/wikipedia/en/wiki/Core_dump
Debugger is for example dbx program

Do you need my help helping the support by explaining what is debugging?
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
davidwang69

ASKER
gheist,

Thanks for your advise!

I have generated the core dump. I need to analyze it. I need to install gdb in our unix box. Any idea what is better tool? I don't have much experience on analyzing core dump since I am a developer. Might need your help on this.
gheist

You could already have dbx on solaris ready.
davidwang69

ASKER
I did a pktrace and output as below:

pstack core.21751
core 'core.21751' of 21751:     /opt/apache/bin/httpd -f /opt/apache/conf/mkb1/httpd
-----------------  lwp# 1 / thread# 1  --------------------
 fee4d6d8 lwp_wait (11, ffbff6b4)
 fee465b8 _thrp_join (11, 0, ffbff71c, 1, ffbff6b4, feeb5900) + 34
 ff1d4f44 apr_thread_join (ffbff79c, 13cf68, 2, 0, 70000, 13cf68) + c
 0002d13c join_workers (54c00, 6a1740, 2cce8, 13cac8, 0, 1) + d0
 0002d468 child_main (f, 2c100, 0, 0, feeb5960, ff1e2a00) + 248
 0002d648 make_child (71000, f, 6, 71400, 7dc78, 70000) + 128
 0002ddcc ap_mpm_run (fe290108, 10, 54ce, 7, 70000, 10) + 694
 000348e0 main     (79f10, 7be98, ffbffaf8, 70000, 70000, 0) + 6a4
 00025fb0 _start   (0, 0, 0, 0, 0, 0) + 5c
-----------------  lwp# 17 / thread# 17  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 8fdda8, ff37ab08, 0, fdee7a00, ff37c29c) + 34
 ff37beb4 apr_dbm_close (140e48, 8fe044, 0, 0, 8fdfac, 57) + c
 fe689fec ssl_scache_dbm_retrieve (121248, 7efd4f, 20, 7a0d0, 12eee8, 0) + 108
 fe689848 ssl_scache_retrieve (121248, 7efd4f, 20, 7efd6f, 8a6460, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (8749b8, 7efd4f, 20, fd07b6f8, fe683298, e) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 7efd4f, 20, 0, 0, b0c) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 32d7d0) + 1c4
 fe626900 ssl3_accept (8749b8, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (8749b8, fc75e698, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (8749b8, 2001, 1, 907db8, b, fe625fa0) + 70
 fe630a24 ssl23_accept (8749b8, fe630800, 8a6370, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (8749b8, 7bc1c0, 8e29c8, 8a64b8, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (8a67c0, 8a6460, 8a6768, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (873618, 8e29e0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 8e29e0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 8e29e0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (8e1d80, 2000, fd07bdb0, 8e1d68, 0, 8e29e0) + 50
 0003e2cc ap_read_request (8a6460, e, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (8a6460, 8a6370, 8a6370, 3ce, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (8a6460, 8a6370, 8a6370, 3ce, 8a6458, 86f5e0) + 3c
 0002cc38 worker_thread (13cf68, e, 0, 70000, 70000, 38) + 1b0
 ff1d4e08 dummy_worker (13cf68, fd07c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 18 / thread# 18  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 19 / thread# 19  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 20 / thread# 20  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 21 / thread# 21  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 22 / thread# 22  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 23 / thread# 23  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 24 / thread# 24  --------------------
 ff1da4fc apr_pool_cleanup_kill (79e90, 81dfa0, ff37ab08, 644, 20, ff37c29c) + 38
 ff37beb4 apr_dbm_close (7ae30, fc97b588, fc97b590, 0, 82b42f, 20) + c
 fe689f8c ssl_scache_dbm_retrieve (121248, 82b42f, 20, 7a0d0, 12eee8, 0) + a8
 fe689848 ssl_scache_retrieve (121248, 82b42f, 20, 82b44f, 81f520, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (45f300, 82b42f, 20, fc97b6f8, fe683298, 2) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 82b42f, 20, 0, 0, c7c8) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 782808) + 1c4
 fe626900 ssl3_accept (45f300, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (45f300, fc14b060, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (45f300, 2001, 1, 7a24e0, b, fe625fa0) + 70
 fe630a24 ssl23_accept (45f300, fe630800, 0, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (45f300, 7bbf98, 82a0b0, 81f578, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (81f880, 81f520, 81f828, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (828440, 82a0c8, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 82a0c8, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 82a0c8, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (829468, 2000, fc97bdb0, 829450, 0, 82a0c8) + 50
 0003e2cc ap_read_request (81f520, 15, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (81f520, 81f430, 81f430, 3d5, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (81f520, 81f430, 81f430, 3d5, 81f518, 821400) + 3c
 0002cc38 worker_thread (13d048, 15, 0, 70000, 70000, 54) + 1b0
 ff1d4e08 dummy_worker (13d048, fc97c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 25 / thread# 25  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 81dfa0, ff37ab08, 0, fdeeba00, ff37c29c) + 34
 ff37beb4 apr_dbm_close (7ae30, 81e184, 0, 0, 81e0ec, 35) + c
 fe689fec ssl_scache_dbm_retrieve (121248, 80992f, 20, 7a0d0, 12eee8, 0) + 108
 fe689848 ssl_scache_retrieve (121248, 80992f, 20, 80994f, 7bc370, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (45f1c0, 80992f, 20, fc87b6f8, fe683298, 1) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 80992f, 20, 0, 0, 8f90) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 6e2ca0) + 1c4
 fe626900 ssl3_accept (45f1c0, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (45f1c0, fc06cb60, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (45f1c0, 2001, 1, 7b3908, b, fe625fa0) + 70
 fe630a24 ssl23_accept (45f1c0, fe630800, 7bc280, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (45f1c0, 7bbf50, 7e29a8, 7bc3c8, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (7bc6d0, 7bc370, 7bc678, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (18ab08, 7e29c0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 7e29c0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 7e29c0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (7e1d60, 2000, fc87bdb0, 7e1d48, 0, 7e29c0) + 50
 0003e2cc ap_read_request (7bc370, 16, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (7bc370, 7bc280, 7bc280, 3d6, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (7bc370, 7bc280, 7bc280, 3d6, 7bc368, 7be250) + 3c
 0002cc38 worker_thread (13d068, 16, 0, 70000, 70000, 58) + 1b0
 ff1d4e08 dummy_worker (13d068, fc87c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 26 / thread# 26  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 28 / thread# 28  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 8fdda8, ff37ab08, 20, ff393c68, ff37c29c) + 34
 ff37beb4 apr_dbm_close (140e48, fc578ef0, fc578ee8, 644, 800, 20) + c
 fe68a2a8 ssl_scache_dbm_store (121248, 91d298, 20, fc578ef8, ffffd7c8, 0) + 234
 fe6897a4 ssl_scache_store (121248, 91d298, 20, 4dd4e259, 91d250, 121308) + 88
 fe683238 ssl_callback_NewSessionCacheEntry (4dd4e259, 91d250, e, fe6831c8, 7fd, 64a0d0) + 70
 fe63f70c ssl_update_cache (79f778, 2, 1, 0, 908bb8, 0) + ac
 fe626a9c ssl3_accept (79f778, 2, 21d0, 21d1, 1, 2171) + afc
 fe63fea4 SSL_accept (79f778, fbc6d8b0, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (79f778, 2001, 1, 8c4f48, b, fe625fa0) + 70
 fe630a24 ssl23_accept (79f778, fe630800, 842f38, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (79f778, 7bc178, 851bc8, 843080, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (843388, 843028, 843330, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (84ff58, 851be0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 851be0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 851be0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (850f80, 2000, fc57bdb0, 850f68, 0, 851be0) + 50
 0003e2cc ap_read_request (843028, 17, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (843028, 842f38, 842f38, 3d7, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (843028, 842f38, 842f38, 3d7, 843020, 844f08) + 3c
 0002cc38 worker_thread (13d0c8, 17, 0, 70000, 70000, 5c) + 1b0
 ff1d4e08 dummy_worker (13d0c8, fc57c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)

pflags core.21751
core 'core.21751' of 21751:     /opt/apache/bin/httpd -f /opt/apache/conf/mkb1/httpd
        data model = _ILP32  flags = ORPHAN|RLC|MSACCT|MSFORK
 /1:    flags = STOPPED|ISTOP|ASLEEP  lwp_wait(0x11,0xffbff6b4)
        why = PR_REQUESTED
        sigmask = 0xffbea007,0x0000fff7
 /17:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /18:   <defunct>
 /19:   <defunct>
 /20:   <defunct>
 /21:   <defunct>
 /22:   <defunct>
 /23:   <defunct>
 /24:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /25:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /26:   <defunct>
 /28:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7

Any idea how to interpret the above ?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
ASKER CERTIFIED SOLUTION
davidwang69

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
davidwang69

ASKER
Problem solved after investigation.