Solved

How to read vmstat on Sun Solaris and what are the key items to test

Posted on 2004-10-26
7,418 Views
Last Modified: 2013-12-05
On Sun Solaris I am running the below vmstat and need to know key items to look for in the output. I know I have serious problems and the system is about to crash when 'id' falls below 10. What other key values do I need to be testing?

----- vmstat output
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m4 m1   in   sy   cs us sy id
 0 0 0 22292328 18225184 51 2005 51 5 4 0 0 7  0  1  6 1680  623 1372 32 37 31
0
Question by:rayskelton
    6 Comments
     
    LVL 1

    Expert Comment

    by:SciGuy
    heres a short desription of each field:

    r     in run queue
    b     blocked for resources I/O, paging
    w     swapped

    swap  amount  of  swap   space   currently   available
    free  size of the free list

    re    page reclaims
    mf    minor faults
    pi    KB paged in
    po   KB paged out
    fr    KB freed
    de  KB anticipated mem shortfall
    sr  pages scanned by clocked alg.

    m*  disk operations per second

    in  interrupts/sec
    sys  syscalls/sec
    cs   constext switches/sec

    us  %user time
    sy   %system/kernel time
    id   %idle time

    idle time isnt very useful for diagnosing a crash.  However, if you're running something that eats up CPU and causing a crash, it might also be using up lots of memory as well... (even then, I dont know how a user application can cause the entire system to crash)
     Whatever the cause, it is not likely because of "id" dropping below 10%.

    You'll need to give more info about what you're running to cause the crash
    0
     
    LVL 38

    Expert Comment

    by:wesly_chen
    /var/adm/messages is the place to look at first when you encounter the crash.

    Wesly
    0
     

    Author Comment

    by:rayskelton
    I am looking at this from a developer of the only application on numerous large Solaris systems, which eats much memory and cpu during peak production periods. This is actually a good problem to have, since it means business is good.  I can always count on serious outages to occure, when the id drops below 10 and have added this check into my monitoring software. I was wanting to look at other crutial items within vmstat. I am a developer and not a sys admin, so attempting to identify whatis  the exact problem at a system level is not my concern. My concern is to give a pre warning to production support before a problem actually occurs. This gives them time to shut down batch servers and potential prevent a problem.  
    0
     
    LVL 38

    Expert Comment

    by:wesly_chen
    Okay, then the "swap free" is another item you might want to watch.
    Usually, when swap free go below certain percentage (3%) and the system start unstable.

    Wesly
    0
     
    LVL 20

    Accepted Solution

    by:
    Solaris crashes only if it runs out storage or system software problem. Even the CPU is running 95%, only the running processes are running slower than, it will not crash Solaris. I suspected the crash at your solaris is due to ran of virtual storage. Analyse the crash dump and you will find out the answer.

    My installation has over 30 production Solaris system and I never have the Solaris crash due to high CPU utlization. We setup the monitoring tools to alert Technical Support whenever the CPU utilization of Solaris is over 90%.  Usually I issue top command to find out which process use most of the CPU. Kill the job if I suspect that the process is using extremely high CPU which slows down the system performance.

    VMSTAT only shows the overall performance and it cannot find out the system hang up problem.  Our installation has over 50 production AIX and they never crash because of the CPU is high.  You need to install monitoring tools such as CA-NSM, BMC Patrol, Candle CCC or EcoTools to automate the computer monitoring.

    Propsed System Health Checking
    1. Run out of virtaul storage
       Check the usage of the swap file alert if it is over 80%
    2. Filesystem corruption
       Monitor /var/adm/ras/message
    3. Non-recovery hardware error such as CPU and memory
       Monitor  /var/adm/ras/messages to alert hardware message. You can get a list of hardware message from Solaris
    4. Filesystem ran of space
      Monitor /var/adm/ras/messages if the usage of root, /tmp and /usr filesystem is over 90%


    0
     
    LVL 38

    Assisted Solution

    by:wesly_chen
    Hi,

       My personal experience with Sun Ultra 80/Enterprise E420R have the crash problem with high-loaded CPU.
    It turns out to be the hardware architecture of the clock bus between CPU and memory has bug on this motherboard design.
    No OS patch can really fix this issue (Solaris 7, 8, 9 are all have the same issue).

       Anyway, monitor the "swap usage" and the "/var partition" is important to avoid crashing or hung-up.

    Wesly
    0

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
    In a recent article here at Experts Exchange (http://www.experts-exchange.com/articles/18880/PaperPort-14-in-Windows-10-A-First-Look.html), I discussed my nine-month sandbox testing of the Windows 10 Technical Preview, specifically with respect to r…
    Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
    Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

    934 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    8 Experts available now in Live!

    Get 1:1 Help Now