We help IT Professionals succeed at work.

cause of apache runaway process

davidwang69
davidwang69 asked
on
284 Views
Last Modified: 2014-03-22
We are using apache 2.0 as our web server running in SunOS  5.10.

From time to time we have runaway process which will comsume huge amount of CPU, something like 30 - 60%.

If we don't restart the apache instance there will be no access to the web page.

How can we find out what cause the runaway process?
Comment
Watch Question

Top Expert 2015

Commented:
Do you use infinite loop in PHP? mod_rewrite? htaccess?

Are you able to attach debugger to the process to see what is inside?

Author

Commented:
We have 2 web servers and running Vignette as content management system on 2 app servers. We don't have PHP and htaccess but we do have a lot mod_rewrite rule defined.

What is debugger, core dump?
Top Expert 2015

Commented:
You mention commercial product - it is a good idea to ask THEIR already paid support to help out.

https://secure.wikimedia.org/wikipedia/en/wiki/Core_dump
Debugger is for example dbx program

Do you need my help helping the support by explaining what is debugging?

Author

Commented:
gheist,

Thanks for your advise!

I have generated the core dump. I need to analyze it. I need to install gdb in our unix box. Any idea what is better tool? I don't have much experience on analyzing core dump since I am a developer. Might need your help on this.
Top Expert 2015

Commented:
You could already have dbx on solaris ready.

Author

Commented:
I did a pktrace and output as below:

pstack core.21751
core 'core.21751' of 21751:     /opt/apache/bin/httpd -f /opt/apache/conf/mkb1/httpd
-----------------  lwp# 1 / thread# 1  --------------------
 fee4d6d8 lwp_wait (11, ffbff6b4)
 fee465b8 _thrp_join (11, 0, ffbff71c, 1, ffbff6b4, feeb5900) + 34
 ff1d4f44 apr_thread_join (ffbff79c, 13cf68, 2, 0, 70000, 13cf68) + c
 0002d13c join_workers (54c00, 6a1740, 2cce8, 13cac8, 0, 1) + d0
 0002d468 child_main (f, 2c100, 0, 0, feeb5960, ff1e2a00) + 248
 0002d648 make_child (71000, f, 6, 71400, 7dc78, 70000) + 128
 0002ddcc ap_mpm_run (fe290108, 10, 54ce, 7, 70000, 10) + 694
 000348e0 main     (79f10, 7be98, ffbffaf8, 70000, 70000, 0) + 6a4
 00025fb0 _start   (0, 0, 0, 0, 0, 0) + 5c
-----------------  lwp# 17 / thread# 17  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 8fdda8, ff37ab08, 0, fdee7a00, ff37c29c) + 34
 ff37beb4 apr_dbm_close (140e48, 8fe044, 0, 0, 8fdfac, 57) + c
 fe689fec ssl_scache_dbm_retrieve (121248, 7efd4f, 20, 7a0d0, 12eee8, 0) + 108
 fe689848 ssl_scache_retrieve (121248, 7efd4f, 20, 7efd6f, 8a6460, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (8749b8, 7efd4f, 20, fd07b6f8, fe683298, e) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 7efd4f, 20, 0, 0, b0c) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 32d7d0) + 1c4
 fe626900 ssl3_accept (8749b8, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (8749b8, fc75e698, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (8749b8, 2001, 1, 907db8, b, fe625fa0) + 70
 fe630a24 ssl23_accept (8749b8, fe630800, 8a6370, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (8749b8, 7bc1c0, 8e29c8, 8a64b8, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (8a67c0, 8a6460, 8a6768, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (873618, 8e29e0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 8e29e0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 8e29e0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (8e1d80, 2000, fd07bdb0, 8e1d68, 0, 8e29e0) + 50
 0003e2cc ap_read_request (8a6460, e, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (8a6460, 8a6370, 8a6370, 3ce, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (8a6460, 8a6370, 8a6370, 3ce, 8a6458, 86f5e0) + 3c
 0002cc38 worker_thread (13cf68, e, 0, 70000, 70000, 38) + 1b0
 ff1d4e08 dummy_worker (13cf68, fd07c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 18 / thread# 18  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 19 / thread# 19  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 20 / thread# 20  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 21 / thread# 21  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 22 / thread# 22  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 23 / thread# 23  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 24 / thread# 24  --------------------
 ff1da4fc apr_pool_cleanup_kill (79e90, 81dfa0, ff37ab08, 644, 20, ff37c29c) + 38
 ff37beb4 apr_dbm_close (7ae30, fc97b588, fc97b590, 0, 82b42f, 20) + c
 fe689f8c ssl_scache_dbm_retrieve (121248, 82b42f, 20, 7a0d0, 12eee8, 0) + a8
 fe689848 ssl_scache_retrieve (121248, 82b42f, 20, 82b44f, 81f520, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (45f300, 82b42f, 20, fc97b6f8, fe683298, 2) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 82b42f, 20, 0, 0, c7c8) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 782808) + 1c4
 fe626900 ssl3_accept (45f300, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (45f300, fc14b060, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (45f300, 2001, 1, 7a24e0, b, fe625fa0) + 70
 fe630a24 ssl23_accept (45f300, fe630800, 0, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (45f300, 7bbf98, 82a0b0, 81f578, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (81f880, 81f520, 81f828, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (828440, 82a0c8, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 82a0c8, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 82a0c8, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (829468, 2000, fc97bdb0, 829450, 0, 82a0c8) + 50
 0003e2cc ap_read_request (81f520, 15, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (81f520, 81f430, 81f430, 3d5, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (81f520, 81f430, 81f430, 3d5, 81f518, 821400) + 3c
 0002cc38 worker_thread (13d048, 15, 0, 70000, 70000, 54) + 1b0
 ff1d4e08 dummy_worker (13d048, fc97c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 25 / thread# 25  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 81dfa0, ff37ab08, 0, fdeeba00, ff37c29c) + 34
 ff37beb4 apr_dbm_close (7ae30, 81e184, 0, 0, 81e0ec, 35) + c
 fe689fec ssl_scache_dbm_retrieve (121248, 80992f, 20, 7a0d0, 12eee8, 0) + 108
 fe689848 ssl_scache_retrieve (121248, 80992f, 20, 80994f, 7bc370, 0) + 70
 fe6832c0 ssl_callback_GetSessionCacheEntry (45f1c0, 80992f, 20, fc87b6f8, fe683298, 1) + 28
 fe642e64 ssl_get_prev_session (64a0d0, 80992f, 20, 0, 0, 8f90) + 2c4
 fe623304 ssl3_get_client_hello (20, 2112, 2120, 0, 4000, 6e2ca0) + 1c4
 fe626900 ssl3_accept (45f1c0, 3004, 21d0, 21d1, 1, 2171) + 960
 fe63fea4 SSL_accept (45f1c0, fc06cb60, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (45f1c0, 2001, 1, 7b3908, b, fe625fa0) + 70
 fe630a24 ssl23_accept (45f1c0, fe630800, 7bc280, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (45f1c0, 7bbf50, 7e29a8, 7bc3c8, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (7bc6d0, 7bc370, 7bc678, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (18ab08, 7e29c0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 7e29c0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 7e29c0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (7e1d60, 2000, fc87bdb0, 7e1d48, 0, 7e29c0) + 50
 0003e2cc ap_read_request (7bc370, 16, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (7bc370, 7bc280, 7bc280, 3d6, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (7bc370, 7bc280, 7bc280, 3d6, 7bc368, 7be250) + 3c
 0002cc38 worker_thread (13d068, 16, 0, 70000, 70000, 58) + 1b0
 ff1d4e08 dummy_worker (13d068, fc87c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 26 / thread# 26  --------------------
 ff1d4dfc dummy_worker(), exit value = 0x00000000
        ** zombie (exited, not detached, not yet joined) **
-----------------  lwp# 28 / thread# 28  --------------------
 ff1da4f8 apr_pool_cleanup_kill (79e90, 8fdda8, ff37ab08, 20, ff393c68, ff37c29c) + 34
 ff37beb4 apr_dbm_close (140e48, fc578ef0, fc578ee8, 644, 800, 20) + c
 fe68a2a8 ssl_scache_dbm_store (121248, 91d298, 20, fc578ef8, ffffd7c8, 0) + 234
 fe6897a4 ssl_scache_store (121248, 91d298, 20, 4dd4e259, 91d250, 121308) + 88
 fe683238 ssl_callback_NewSessionCacheEntry (4dd4e259, 91d250, e, fe6831c8, 7fd, 64a0d0) + 70
 fe63f70c ssl_update_cache (79f778, 2, 1, 0, 908bb8, 0) + ac
 fe626a9c ssl3_accept (79f778, 2, 21d0, 21d1, 1, 2171) + afc
 fe63fea4 SSL_accept (79f778, fbc6d8b0, ffffffff, 3, 0, 2000) + 24
 fe630090 ssl23_get_client_hello (79f778, 2001, 1, 8c4f48, b, fe625fa0) + 70
 fe630a24 ssl23_accept (79f778, fe630800, 842f38, fe630800, fe664fc4, fe664fc4) + 224
 fe63fea4 SSL_accept (79f778, 7bc178, 851bc8, 843080, 30, 121308) + 24
 fe67f7cc ssl_io_filter_connect (843388, 843028, 843330, 0, 0, 1) + 8c
 fe67fbf8 ssl_io_filter_input (84ff58, 851be0, 1, 0, 0, fe67fb78) + 80
 0003c5c0 ap_get_brigade (fffffffe, 851be0, 1, 0, 0, 44090) + 30
 0003c5c0 ap_get_brigade (fffffffe, 851be0, 1, 0, 0, 1ec) + 30
 0003d738 ap_rgetline_core (850f80, 2000, fc57bdb0, 850f68, 0, 851be0) + 50
 0003e2cc ap_read_request (843028, 17, 3, 0, 71000, 2) + 184
 00026530 ap_process_http_connection (843028, 842f38, 842f38, 3d7, 71000, 12ff10) + 9c
 00039fd0 ap_run_process_connection (843028, 842f38, 842f38, 3d7, 843020, 844f08) + 3c
 0002cc38 worker_thread (13d0c8, 17, 0, 70000, 70000, 5c) + 1b0
 ff1d4e08 dummy_worker (13d0c8, fc57c000, 0, 0, ff1d4dfc, 1) + c
 fee499b8 _lwp_start (0, 0, 0, 0, 0, 0)

pflags core.21751
core 'core.21751' of 21751:     /opt/apache/bin/httpd -f /opt/apache/conf/mkb1/httpd
        data model = _ILP32  flags = ORPHAN|RLC|MSACCT|MSFORK
 /1:    flags = STOPPED|ISTOP|ASLEEP  lwp_wait(0x11,0xffbff6b4)
        why = PR_REQUESTED
        sigmask = 0xffbea007,0x0000fff7
 /17:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /18:   <defunct>
 /19:   <defunct>
 /20:   <defunct>
 /21:   <defunct>
 /22:   <defunct>
 /23:   <defunct>
 /24:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /25:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7
 /26:   <defunct>
 /28:   flags = STOPPED|ISTOP
        why = PR_REQUESTED
        sigmask = 0xffbee007,0x0000fff7

Any idea how to interpret the above ?
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Problem solved after investigation.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.