How to monitor a production java process running on linux

I need to monitor a java process. At present I monitor the cpu and memory used by it using top command.
And then for that process_id I log the output of
1. lsof -p ${process_pid}
2. pstack -p ${process_pid}
3. top -n 1 -b -p ${process_pid} -H

But these are not giving me any useful results together. How do i collate useful details from them?

I am aware of some useful tools inside jdk folder, but i cant use them as they are not part of the production server.
Is there any detail provided by jvm and linux OS other than what I have listed above?
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Sharon SethCommented:
What do you need to monitor , apart from cpu and memory?
pvinodpAuthor Commented:
which jar is taking how much memory. what is the memory occupied by a class.
Same details for cpu as well.
Radek BaranowskiFull-stack Java DeveloperCommented:
commandline/linux tools won't give you that. the only way is to analyze thread dump or use jmx console wchich is a part of JDK (not JRE)
(binary util jconsole in <jdkhome>/bin)
I'm not sure though if there's any way to use it in batch mode, I doubt it, so there's only way to monitor it visually)
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

pvinodpAuthor Commented:
i am well aware of tools within jdk.
I need to know if there is any help available from jre or linux OS
Radek BaranowskiFull-stack Java DeveloperCommented:

I googled a bit and found something like that:

yet this yields stats you want (I guess) but only AFTERWARDS I mean, the execution needs to end to generate java.hprof.txt file..

more options described here:

pvinodpAuthor Commented: This link suggest that I start the java process with the arguments. But i need to sniff the production java process which has already started and giving trouble. Moreover jhat is part of jre and not jdk.

Even the second link requires similar process.

let me know if I have wrongly understood.
Radek BaranowskiFull-stack Java DeveloperCommented:
yes, possibly.

well what you are trying to do can only be achieved by either jconsole or dumping heap/thread dumps with kill -3 <pid> and analyzing it with available tools (Java Heap/Thread dump analyzer from IBM or write parser of your own)
Performance Monitoring Infrastructure (PMI)

Use this page to learn about Performance Monitoring Infrastructure and other tools to help monitor the overall health of the application server.

A typical Web system consists of a Web server, application server, and a database. Monitoring and tuning the application server is critical to the overall performance of the Web system. Performance Monitoring Infrastructure (PMI) is the core monitoring infrastructure for WebSphere Application Server and WebSphere family products like Portal, Commerce, and so on. The performance data provided by WebSphere PMI helps to monitor and tune the application server performance.

When tuning the WebSphere Application Server for optimal performance, or fixing a poorly performing Java 2 Platform, Enterprise Edition (J2EE) application, it is important to understand how the various run time and application resources are behaving from a performance perspective. PMI provides a comprehensive set of data that explains the runtime and application resource behavior. For example, PMI provides database connection pool size, servlet response time, Enterprise JavaBeans (EJB) method response time, Java virtual machine (JVM) garbage collection time, CPU usage, and so on. This data can be used to understand the runtime resource utilization patterns of the thread pool, connection pool, and so on, and the performance characteristics of the application components like servlets, JavaServer Pages (JSP), and enterprise beans.

Using PMI data, the performance bottlenecks in the application server can be identified and fixed. For instance, one of the PMI statistics in the Java DataBase Connectivity (JDBC) connection pool is the number of statements discarded from prepared statement cache. This statistic can be used to adjust the prepared statement cache size in order to minimize the discards and to improve the database query performance. PMI data can be monitored and analyzed by Tivoli Performance Viewer (TPV), other Tivoli tools, your own applications, or third party tools. TPV is a graphical viewer for PMI data that ships with WebSphere Application Server. Performance advisors use PMI data to analyze the run-time state of the application server, and provide tuning advice to optimize the application server resource utilization.

PMI data can also be used to monitor the health of the application server. Some of the health indicators are CPU usage, Servlet response time, and JDBC query time. Performance management tools like Tivoli Monitoring for Web Infrastructure and other third party tools can monitor the PMI data and generate alerts based on some predefined thresholds.

Monitoring system performance using WebSphere PMI
If a java process is already started with non-instrumentation options, there is very little we can do. However, in practice, i found the following handy and often helps me figure out most the the issues:

1. Take thread-dump: java -QUIT <pid> prints the thread-dump on the console. Useful in working with locks, dead-locks and application flows.
2. Heap dump, use jmap to dump the heap. It gives insights of the objects, leaks
3. jstat - prints stat information of running jvm.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
pvinodpAuthor Commented:
thats a lot useful information. thanks all
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.