Link to home
Create AccountLog in
Avatar of Vlearns
Vlearns

asked on

java performance debugging tips

Hi

we have a java application inside a jvm running on a box.

after sometime, the cpu hits 99% mostly in user space

top - 02:51:32 up 6 days, 20:54,  6 users,  load average: 13.70, 12.47, 12.15
Tasks: 402 total,   1 running, 397 sleeping,   4 stopped,   0 zombie
Cpu(s): 99.4%us,  0.5%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  65321916k total, 43375196k used, 21946720k free,   593432k buffers
top - 02:51:57 up 6 days, 20:55,  7 users,  load average: 13.26, 12.46, 12.16
Tasks: 403 total,   1 running, 398 sleeping,   4 stopped,   0 zombie
Cpu(s): 98.1%us,  0.8%sy,  0.0%ni,  0.8%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  65321916k total, 43375968k used, 21945948k free,   593432k buffers
Swap: 12582904k total,        0k used, 12582904k free,  3351852k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 15761 aho    20   0 38.9g  34g  37m S 788.4 55.0 668:07.41 java_daemon
477671 aho     20   0 19.9g 489m  14m S  2.0  0.8  22:35.56 java
477149 root      20   0  126m  58m  23m S  1.3  0.1   4:01.88 hmproxy

     9 root      20   0     0    0    0 S  0.3  0.0  40:53.16 ksoftirqd/1

a pastack on the pid java_daemon

Thread 952 (Thread 0x7f6e08272700 (LWP 15763)):^M
#0  0x00000036bf60b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0^M
#1  0x00007f76193fb793 in os::PlatformEvent::park() () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#2  0x00007f76193c759f in Monitor::IWait(Thread*, long) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#3  0x00007f76193c7d2e in Monitor::wait(bool, long, bool) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#4  0x00007f76191719f3 in GCTaskManager::get_task(unsigned int) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#5  0x00007f761917257d in GCTaskThread::run() () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#6  0x00007f7619401c00 in java_start(Thread*) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#7  0x00000036bf607851 in start_thread () from /lib64/libpthread.so.0^M
#8  0x00000036bf2e76dd in clone () from /lib64/libc.so.6^M
Thread 951 (Thread 0x7f6e08171700 (LWP 15764)):^M
#0  0x00000036bf60b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0^M
#1  0x00007f76193fb793 in os::PlatformEvent::park() () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#2  0x00007f76193c759f in Monitor::IWait(Thread*, long) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#3  0x00007f76193c7d2e in Monitor::wait(bool, long, bool) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#4  0x00007f76191719f3 in GCTaskManager::get_task(unsigned int) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#5  0x00007f761917257d in GCTaskThread::run() () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#6  0x00007f7619401c00 in java_start(Thread*) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#7  0x00000036bf607851 in start_thread () from /lib64/libpthread.so.0^M
#8  0x00000036bf2e76dd in clone () from /lib64/libc.so.6^M
Thread 950 (Thread 0x7f6e08070700 (LWP 15765)):^M



Thread 876 (Thread 0x7f6df47f6700 (LWP 16054)):^M
#0  0x00000036bf2e7cd3 in epoll_wait () from /lib64/libc.so.6^M
#1  0x00007f6e05136739 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /home/y/libexec64/jdk1.7.0/jre/lib/amd64/libnio.so^M
#2  0x00007f76153eddba in ?? ()^M
#3  0x00007f6e1ec48998 in ?? ()^M
#4  0x00007f6d58337c18 in ?? ()^M
#5  0x0000000000000007 in ?? ()^M
#6  0x0000000002157070 in ?? ()^M
#7  0x00007f6df47f54c0 in ?? ()^M
#8  0x00007f7619058d05 in CollectedHeap::new_store_pre_barrier(JavaThread*, oopDesc*) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#9  0x00007f761547edd4 in ?? ()^M
#10 0x000000000000000a in ?? ()^M
#11 0x00007f6e1ec48998 in ?? ()^M
#12 0x00007f76199c8a70 in Universe::_heap_used_at_last_gc () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#13 0x0000000000000000 in ?? ()^M
Thread 875 (Thread 0x7f6df46f5700 (LWP 16055)):^M
#0  0x00000036bf2e7cd3 in epoll_wait () from /lib64/libc.so.6^M
#1  0x00007f6e05136739 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /home/y/libexec64/jdk1.7.0/jre/lib/amd64/libnio.so^M
#2  0x00007f76153eddba in ?? ()^M
#3  0x00007f6e1e938fa0 in ?? ()^M
#4  0x00007f6d5487fff8 in ?? ()^M
#5  0x0000000000000007 in ?? ()^M
#6  0x0000000002157070 in ?? ()^M
#7  0x00007f6df46f4550 in ?? ()^M
#8  0x00007f7619058d05 in CollectedHeap::new_store_pre_barrier(JavaThread*, oopDesc*) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#9  0x00007f761547edd4 in ?? ()^M
#10 0x000000000000000a in ?? ()^M
#11 0x00007f6e1e938fa0 in ?? ()^M
#12 0x00007f76199c8a70 in Universe::_heap_used_at_last_gc () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#13 0x0000000000000000 in ?? ()^M
Thread 874 (Thread 0x7f6df45f4700 (LWP 16056)):^M
#0  0x00000036bf2e7cd3 in epoll_wait () from /lib64/libc.so.6^M
#1  0x00007f6e05136739 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /home/y/libexec64/jdk1.7.0/jre/lib/amd64/libnio.so^M
#2  0x00007f76153eddba in ?? ()^M
#3  0x00007f6e219e8110 in ?? ()^M
#4  0x00007f6d54853468 in ?? ()^M
#5  0x0000000000000007 in ?? ()^M
#6  0x0000000002157070 in ?? ()^M
#7  0x00007f6df45f33c0 in ?? ()^M
#8  0x00007f7619058d05 in CollectedHeap::new_store_pre_barrier(JavaThread*, oopDesc*) () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M
#9  0x00007f761547edd4 in ?? ()^M
#10 0x000000000000000a in ?? ()^M
#11 0x00007f6e219e8110 in ?? ()^M
#12 0x00007f76199c8a70 in Universe::_heap_used_at_last_gc () from /home/y/share/yjava_jdk/java/jre/lib/amd64/server/libjvm.so^M

~                                                                        

any ideas on how to debug this issue...
SOLUTION
Avatar of basav_com
basav_com
Flag of India image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
SOLUTION
Avatar of krakatoa
krakatoa
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of Vlearns
Vlearns

ASKER

Thanks for the feedback
does anything in the trace i posted come across as a problem?
Can't really say about that to any extent, my points amounting to a couple of obvious generalities.
ASKER CERTIFIED SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Doug's comments carry a lot more weight than my stabs.

Now that you mention it though:

does anything in the trace i posted come across as a problem?

you have got what . . .  875-odd threads running? Isn't that a lot of threads?