Link to home
Start Free TrialLog in
Avatar of davidwang69
davidwang69

asked on

cause of apache runaway process

We are using apache 2.0 as our web server running in SunOS  5.10.

From time to time we have runaway process which will comsume huge amount of CPU, something like 30 - 60%.

If we don't restart the apache instance there will be no access to the web page.

How can we find out what cause the runaway process?
Avatar of gheist
gheist
Flag of Belgium image

Do you use infinite loop in PHP? mod_rewrite? htaccess?

Are you able to attach debugger to the process to see what is inside?
Avatar of davidwang69
davidwang69

ASKER

We have 2 web servers and running Vignette as content management system on 2 app servers. We don't have PHP and htaccess but we do have a lot mod_rewrite rule defined.

What is debugger, core dump?
You mention commercial product - it is a good idea to ask THEIR already paid support to help out.

https://secure.wikimedia.org/wikipedia/en/wiki/Core_dump
Debugger is for example dbx program

Do you need my help helping the support by explaining what is debugging?
gheist,

Thanks for your advise!

I have generated the core dump. I need to analyze it. I need to install gdb in our unix box. Any idea what is better tool? I don't have much experience on analyzing core dump since I am a developer. Might need your help on this.
You could already have dbx on solaris ready.
I did a pktrace and output as below:

pstack core.21751
core 'core.21751' of 21751:     /opt/apache/bin/httpd -f /opt/apache/conf/mkb1/httpd
-----------------  lwp# 1 / thread# 1  --------------------
 fee4d6d8 lwp_wait (11, ffbff6b4)
 fee465b8 _thrp_join (11, 0, ffbff71c, 1, ffbff6b4, feeb5900) + 34
 ff1d4f44 apr_thread_join (ffbff79c, 13cf68, 2, 0, 70000, 13cf68) + c
 0002d13c join_workers (54c00, 6a1740, 2cce8, 13cac8, 0, 1) + d0
 0002d468 child_main (f, 2c100, 0, 0, feeb5960, ff1e2a00) + 248
 0002d648 make_child (71000, f, 6, 71400, 7dc78, 70000) + 128
 0002ddcc ap_mpm_run (fe290108, 10, 54ce, 7, 70000, 10) + 694
 000348e0 main     (79f10, 7be98, ffbffaf8, 70000, 70000, 0) + 6a4
 00025fb0 _start   (0, 0, 0, 0, 0, 0) + 5c
-----------------  lwp# 17 / thread# 17  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 8fdda8, ff37ab08, 0, fdee7a00, ff37c29c) + 34
 ff37beb4 apr_dbm_close (140e48, 8fe044, 0, 0, 8fdfac, 57) + c
 fe689fec ssl_scache_dbm_retrieve (121248, 7efd4f, 20, 7a0d0, 12eee8, 0) + 108
 fe689848 ssl_scache_retrieve (121248, 7efd4f, 20, 7efd6f, 8a6460, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (8749b8, 7efd4f, 20, fd07b6f8, fe683298, e) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 7efd4f, 20, 0, 0, b0c) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 32d7d0) + 1c4
 fe626900 ssl3_accept (8749b8, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (8749b8, fc75e698, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (8749b8, 2001, 1, 907db8, b, fe625fa0) + 70
 fe630a24 ssl23_accept (8749b8, fe630800, 8a6370, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (8749b8, 7bc1c0, 8e29c8, 8a64b8, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (8a67c0, 8a6460, 8a6768, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (873618, 8e29e0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 8e29e0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 8e29e0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (8e1d80, 2000, fd07bdb0, 8e1d68, 0, 8e29e0) + 50
 0003e2cc ap_read_request (8a6460, e, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (8a6460, 8a6370, 8a6370, 3ce, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (8a6460, 8a6370, 8a6370, 3ce, 8a6458, 86f5e0) + 3c
 0002cc38 worker_thread (13cf68, e, 0, 70000, 70000, 38) + 1b0
 ff1d4e08 dummy_worker (13cf68, fd07c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 18 / thread# 18  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 19 / thread# 19  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 20 / thread# 20  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 21 / thread# 21  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 22 / thread# 22  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 23 / thread# 23  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 24 / thread# 24  --------------------
 ff1da4fc apr_pool_cleanup_kill (79e90, 81dfa0, ff37ab08, 644, 20, ff37c29c) + 38
 ff37beb4 apr_dbm_close (7ae30, fc97b588, fc97b590, 0, 82b42f, 20) + c
 fe689f8c ssl_scache_dbm_retrieve (121248, 82b42f, 20, 7a0d0, 12eee8, 0) + a8
 fe689848 ssl_scache_retrieve (121248, 82b42f, 20, 82b44f, 81f520, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (45f300, 82b42f, 20, fc97b6f8, fe683298, 2) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 82b42f, 20, 0, 0, c7c8) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 782808) + 1c4
 fe626900 ssl3_accept (45f300, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (45f300, fc14b060, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (45f300, 2001, 1, 7a24e0, b, fe625fa0) + 70
 fe630a24 ssl23_accept (45f300, fe630800, 0, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (45f300, 7bbf98, 82a0b0, 81f578, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (81f880, 81f520, 81f828, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (828440, 82a0c8, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 82a0c8, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 82a0c8, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (829468, 2000, fc97bdb0, 829450, 0, 82a0c8) + 50
 0003e2cc ap_read_request (81f520, 15, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (81f520, 81f430, 81f430, 3d5, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (81f520, 81f430, 81f430, 3d5, 81f518, 821400) + 3c
 0002cc38 worker_thread (13d048, 15, 0, 70000, 70000, 54) + 1b0
 ff1d4e08 dummy_worker (13d048, fc97c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 25 / thread# 25  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 81dfa0, ff37ab08, 0, fdeeba00, ff37c29c) + 34
 ff37beb4 apr_dbm_close (7ae30, 81e184, 0, 0, 81e0ec, 35) + c
 fe689fec ssl_scache_dbm_retrieve (121248, 80992f, 20, 7a0d0, 12eee8, 0) + 108
 fe689848 ssl_scache_retrieve (121248, 80992f, 20, 80994f, 7bc370, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (45f1c0, 80992f, 20, fc87b6f8, fe683298, 1) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 80992f, 20, 0, 0, 8f90) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 6e2ca0) + 1c4
 fe626900 ssl3_accept (45f1c0, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (45f1c0, fc06cb60, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (45f1c0, 2001, 1, 7b3908, b, fe625fa0) + 70
 fe630a24 ssl23_accept (45f1c0, fe630800, 7bc280, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (45f1c0, 7bbf50, 7e29a8, 7bc3c8, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (7bc6d0, 7bc370, 7bc678, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (18ab08, 7e29c0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 7e29c0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 7e29c0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (7e1d60, 2000, fc87bdb0, 7e1d48, 0, 7e29c0) + 50
 0003e2cc ap_read_request (7bc370, 16, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (7bc370, 7bc280, 7bc280, 3d6, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (7bc370, 7bc280, 7bc280, 3d6, 7bc368, 7be250) + 3c
 0002cc38 worker_thread (13d068, 16, 0, 70000, 70000, 58) + 1b0
 ff1d4e08 dummy_worker (13d068, fc87c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 26 / thread# 26  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 28 / thread# 28  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 8fdda8, ff37ab08, 20, ff393c68, ff37c29c) + 34
 ff37beb4 apr_dbm_close (140e48, fc578ef0, fc578ee8, 644, 800, 20) + c
 fe68a2a8 ssl_scache_dbm_store (121248, 91d298, 20, fc578ef8, ffffd7c8, 0) + 234
 fe6897a4 ssl_scache_store (121248, 91d298, 20, 4dd4e259, 91d250, 121308) + 88
 fe683238 ssl_callback_NewSessionCacheEntry (4dd4e259, 91d250, e, fe6831c8, 7fd, 64a0d0) + 70
 fe63f70c ssl_update_cache (79f778, 2, 1, 0, 908bb8, 0) + ac
 fe626a9c ssl3_accept (79f778, 2, 21d0, 21d1, 1, 2171) + afc
 fe63fea4 SSL_accept (79f778, fbc6d8b0, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (79f778, 2001, 1, 8c4f48, b, fe625fa0) + 70
 fe630a24 ssl23_accept (79f778, fe630800, 842f38, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (79f778, 7bc178, 851bc8, 843080, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (843388, 843028, 843330, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (84ff58, 851be0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 851be0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 851be0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (850f80, 2000, fc57bdb0, 850f68, 0, 851be0) + 50
 0003e2cc ap_read_request (843028, 17, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (843028, 842f38, 842f38, 3d7, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (843028, 842f38, 842f38, 3d7, 843020, 844f08) + 3c
 0002cc38 worker_thread (13d0c8, 17, 0, 70000, 70000, 5c) + 1b0
 ff1d4e08 dummy_worker (13d0c8, fc57c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)

pflags core.21751
core 'core.21751' of 21751:     /opt/apache/bin/httpd -f /opt/apache/conf/mkb1/httpd
        data model = _ILP32  flags = ORPHAN|RLC|MSACCT|MSFORK
 /1:    flags = STOPPED|ISTOP|ASLEEP  lwp_wait(0x11,0xffbff6b4)
        why = PR_REQUESTED
        sigmask = 0xffbea007,0x0000fff7
 /17:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /18:   <defunct>
 /19:   <defunct>
 /20:   <defunct>
 /21:   <defunct>
 /22:   <defunct>
 /23:   <defunct>
 /24:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /25:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /26:   <defunct>
 /28:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7

Any idea how to interpret the above ?
ASKER CERTIFIED SOLUTION
Avatar of davidwang69
davidwang69

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Problem solved after investigation.