Solved

how to analyse apache web server core dump

Posted on 2011-02-18
18
1,429 Views
Last Modified: 2012-05-11
i used gbd to analyse the core dump produced by my apache (http) web server, i get below output. How to analyse code dump files produced by OS. Mine is Redhat Linux 32 bit OS on Intel x86 hardware.


(gdb) core core.22226
Core was generated by `/opt/covalent/ers-4.0.0-1/apache2.2/bin/httpsd.worker -f /user/local/app/ssl_conf'.
Program terminated with signal 11, Segmentation fault.
[New process 22226]
[New process 22256]
[New process 22255]
[New process 22253]
[New process 22252]
[New process 22251]
[New process 22250]
[New process 22249]
[New process 22248]
[New process 22247]
[New process 22246]
[New process 22245]
[New process 22244]
[New process 22243]
[New process 22242]
[New process 22241]
[New process 22240]
[New process 22239]
[New process 22238]
[New process 22237]
[New process 22236]
[New process 22235]
[New process 22234]
[New process 22233]
[New process 22232]
[New process 22231]
[New process 22230]
[New process 22229]
[New process 22228]
#0  0x0056c7a2 in ?? ()
(gdb)

The above output what does it say
0
Comment
Question by:wasman
  • 8
  • 7
  • 3
18 Comments
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
You need to enter gdb /opt/covalent/ers-4.0.0-1/apache2.2/bin/httpsd.worker core.22226 so that gdb can load up the symbols and references to shared libraries.
Try that and post back with results
0
 
LVL 9

Assisted Solution

by:svs
svs earned 100 total points
Comment Utility
The above output says that gdb successfully loaded a core file and awaits commands.  Try these:
where
thread apply all bt

Open in new window

0
 

Author Comment

by:wasman
Comment Utility
After i provide  ---> thread apply all bt, i get below output, i couldn't understand what caused core dump looking at below output

Thread 29 (process 22358):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x005adcc6 in do_sigwaitinfo () from /lib/tls/libc.so.6
No symbol table info available.
#2  0x0070fa98 in sigwait () from /lib/tls/libpthread.so.0
No symbol table info available.
#3  0x001d535b in std::string::_Rep::_M_grab () from /usr/lib/libstdc++.so.5
No symbol table info available.
#4  0x001d546c in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string () from /usr/lib/libstdc++.so.5
No symbol table info available.
#5  0x0256fe04 in CSmHttpPlugin::ResolveAgentName () from /opt/app/siteminder/66/webagent/lib/libHttpPlugin.so
No symbol table info available.
#6  0x0254e5a7 in CSmHttpPlugin::ProcessResource () from /opt/app/siteminder/66/webagent/lib/libHttpPlugin.so
No symbol table info available.
#7  0x002395ed in CSmResourceManager::ProcessResource () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#8  0x00231731 in CSmHighLevelAgent::ProcessRequest () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#9  0x00228810 in sm_access_checker () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#10 0x080741b8 in ap_run_access_checker ()
No symbol table info available.
#11 0x0807480e in ap_process_request_internal ()
No symbol table info available.
#12 0x08088548 in ap_process_request ()
No symbol table info available.
#13 0x08085108 in ap_process_http_connection ()
No symbol table info available.
#14 0x08080b9d in ap_run_process_connection ()
No symbol table info available.
#15 0x08080f88 in ap_process_connection ()
No symbol table info available.
#16 0x08090fe4 in worker_thread ()
No symbol table info available.
#17 0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#18 0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#19 0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 28 (process 22359):
#0  0x002d79a2 in CSmServerHandle::ProcessRequest () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#1  0x002db64b in CSmCluster::ProcessRequest () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#2  0x0029c1a0 in ProcessRequest () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#3  0x002aeb5c in Sm_AgentApi_AuthorizeEx () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#4  0x0026e55c in SmAuthorizationCache::AuthorizeEx () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#5  0x00248b48 in CSmLowLevelAgent::AuthorizeUser () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#6  0x002363b0 in CSmAuthorizationManager::AuthorizeUser () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#7  0x00232758 in CSmHighLevelAgent::ProcessRequest () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#8  0x00228810 in sm_access_checker () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#9  0x080741b8 in ap_run_access_checker ()
No symbol table info available.
#10 0x0807480e in ap_process_request_internal ()
No symbol table info available.
#11 0x08088548 in ap_process_request ()
No symbol table info available.
#12 0x08085108 in ap_process_http_connection ()
No symbol table info available.
#13 0x08080b9d in ap_run_process_connection ()
No symbol table info available.
#14 0x08080f88 in ap_process_connection ()
No symbol table info available.
#15 0x08090fe4 in worker_thread ()
No symbol table info available.
#16 0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#17 0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#18 0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 27 (process 22360):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 26 (process 22361):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 25 (process 22362):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 24 (process 22363):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 23 (process 22364):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 22 (process 22365):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 21 (process 22366):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 20 (process 22367):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 19 (process 22368):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 18 (process 22369):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 17 (process 22370):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 16 (process 22371):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 15 (process 22372):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 14 (process 22373):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 13 (process 22374):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 12 (process 22375):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 11 (process 22376):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 10 (process 22377):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 9 (process 22378):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 8 (process 22379):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 7 (process 22380):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 6 (process 22381):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 5 (process 22382):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070bcf6 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x08091552 in ap_queue_pop ()
No symbol table info available.
#3  0x08090e4d in worker_thread ()
No symbol table info available.
#4  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#5  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#6  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 4 (process 22383):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070e8a8 in __read_nocancel () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0xed7ff2d8 in ?? ()
No symbol table info available.
#3  0x00b8498f in apr_socket_accept () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#4  0x0809213c in unixd_accept ()
No symbol table info available.
#5  0x080906b8 in listener_thread ()
No symbol table info available.
#6  0x00b891e6 in dummy_worker () from /opt/covalent/4.0/apache2.2/lib/libapr-1.so.0
No symbol table info available.
#7  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#8  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 3 (process 22385):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0064a3b1 in ptrace () from /lib/tls/libc.so.6
No symbol table info available.
#2  0x0029c892 in ConnectionService () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#3  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 2 (process 22386):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0064a3b1 in ptrace () from /lib/tls/libc.so.6
No symbol table info available.
#2  0x0023f26b in CSmAdminManager::ManageAgentThread () from /opt/app/siteminder/66/webagent/lib/libmod_sm22.so
No symbol table info available.
#3  0x007093cc in __nptl_deallocate_tsd () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0x0065196e in query_module () from /lib/tls/libc.so.6
No symbol table info available.

Thread 1 (process 22356):
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
No symbol table info available.
#1  0x0070e69b in __lll_mutex_timedlock_wait () from /lib/tls/libpthread.so.0
No symbol table info available.
#2  0x080a8d0c in __JCR_LIST__ ()
No symbol table info available.
#3  0x00000000 in ?? ()
No symbol table info available.
(gdb)
0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
Did you start gdb as I posted in http:#a34946272 ? There is no symbol information in core files.
0
 

Author Comment

by:wasman
Comment Utility
yes i started it like below
$gdb /opt/covalent/4.0/apache2.2/bin/httpsd.worker /tmp/core.22356
0
 
LVL 9

Expert Comment

by:svs
Comment Utility
I'd say that it crashed in CSmHttpPlugin::ResolveAgentName () from /opt/app/siteminder/66/webagent/lib/libHttpPlugin.so

Output from "where" command is needed to confirm that.
0
 

Author Comment

by:wasman
Comment Utility


(gdb) where
#0  0x0056c7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x0070e69b in __lll_mutex_timedlock_wait () from /lib/tls/libpthread.so.0
#2  0x080a8d0c in __JCR_LIST__ ()
#3  0x00000000 in ?? ()
(gdb) thread apply all bt
0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
@svs
Why do you finger CSmHttpPlugin::ResolveAgentName? That thread is waiting on a signal. The only thread that is doing anything other than waiting in _dl_sysinfo_int80() is thread 28 in CSmServerHandle::ProcessRequest ().
Without a symbol table though, there's not much information to be had.
@wasman
This stack trace shows the old Linux threads library being used. What revision of libc are you running? Please post the output from /lib/libc.so.6 - that will tell us.
0
 

Author Comment

by:wasman
Comment Utility
$ /lib/libc.so.6
GNU C Library stable release version 2.3.4, by Roland McGrath et al.
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.4.6 20060404 (Red Hat 3.4.6-11).
Compiled on a Linux 2.4.20 system on 2010-03-24.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        linuxthreads-0.10 by Xavier Leroy
        The C stubs add-on version 2.1.2.
        BIND-8.2.3-T5B
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        Glibc-2.0 compatibility add-on by Cristian Gafton
        GNU Libidn by Simon Josefsson
        libthread_db work sponsored by Alpha Processor Inc
Thread-local storage support included.
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
Are you running Linux 2.4? (uname -r will show you). Not sure how relevant that is, just interested to see such old software still in use.
Now to apache - is it failing continuously, or is this core dump from a one-off event?
If it keeps happening, you might do well to install libraries with debug symbolic information (not a standard thing to do). This core dump is not going to tell you a whole lot more than it has.
0
 

Author Comment

by:wasman
Comment Utility
$ cat /etc/*release
Red Hat Enterprise Linux AS release 4 (Nahant Update 7)
0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
Sorry that tells me nothing. I asked for a Linux revision number, not a Distributor release number. Red Hat is just a distribution. uname -r will report Linux revision
0
 
LVL 9

Expert Comment

by:svs
Comment Utility
RHEL4 normally uses 2.6.9 kernel.

The backtrace actually shows that NPTL thread library is in use (/lib/tls/libc.so.6).
0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
@svs
tls = Thread Local Storage. The threads show as processes, not threads. The output from /lib/libc.so.6 shows linuxthreads-0.10 by Xavier Leroy whereas NPTL threads require Native POSIX Threads Library by Ulrich Drepper et al.
GNU C Library stable release version 2.3.4 can be built to support Native POSIX Threads, but not when built under Linux 2.4 as this one was. The first NPTL version was 2.3.3 under Linux 2.6 (2.6 is required for NPTL).
@wasman
Please post the output from uname -r
0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
@wasman
Is the httpsd.worker crash a regular occurrence or was it a one-off?
0
 

Author Comment

by:wasman
Comment Utility
sorry for delay
$ uname -r
2.6.9-78.0.8.ELhugemem

core dumps are coming frequently. i would say two in a week.
0
 
LVL 34

Accepted Solution

by:
Duncan Roe earned 400 total points
Comment Utility
I think this is likely to be a rogue script somewhere in your site. Have you got logs of pages that are accessed? You might be able to check whether the same one shows up near every crash.
My concern about analysing core dumps is that it's going to take a lot of work on your part to install debug versions of shared objects and executables and even if you do get the source line where the crash occurs, you may not get a pointer back to the rogue script. Of course you will have found a bug, and if you can then fix it so that apache doesn't crash but instead returns a message to the user; then the user might contact you and you will find out what he was doing.
It seems to me if you rebuild /opt/app/siteminder/66/webagent/lib/libmod_sm22.so and /opt/covalent/ers-4.0.0-1/apache2.2/bin/httpsd.worker with full debug (-g3 -ggdb) and no optimising then you should get a fully usable backtrace. In such a backtrace you can move up and down frames and examine variables, so you should be able to work out what has gone wrong.
Are you comfortable with rebuilding?
(I know gdb can be used with optimised code, but you get more information and less confusion if it's not optimised)
0
 

Author Closing Comment

by:wasman
Comment Utility
Thansks a lot Duncan and svs for your efforts. I will take your suggestions to improve my env.
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

Suggested Solutions

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now