Solved

AIX 5.3 vmstat iostat output, signs of a overloaded system?

Posted on 2010-11-15
4
1,239 Views
Last Modified: 2012-06-27
Hi,

New to the AIX world here, we are running AIX 5.3 64BIT ML-9. Below is a sample output of the various performance monitoring tools:

vmstat 1 10
procs                      memory      swap          io     system         cpu
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  5 375912  19548  17556 477472    0    1     0     0    1     1  1  0  0  1
 0  4 375912  18700  17556 478264    0    0  1044     0  774  1329  8  1  0 91
 0  5 375912  17664  17556 479168    0    0  1160     0  764  1110  8  1  0 91
 1  8 375912  15836  17568 479796    0    0  1144   840  751  1622 16  7  0 78
 0  7 375912  19340  17576 480224    0    0  1224   148  587  1958 17 18  0 65
 2  0 375912  18288  17588 481036    0    0   812     0  845  1732 18  3 21 59
 0  2 375912  15868  17588 481528    0    0  1012     0  588   941  4  1  5 90 

Open in new window


sar -u 2 3
AIX van-oppy 3 5 00001501D600    11/15/10

System configuration: lcpu=8  mode=Capped 

17:26:01    %usr    %sys    %wio   %idle   physc
17:26:03      17       8       9      66    3.98
17:26:05      14       4       6      76    4.00
17:26:07      19       4       2      75    4.00

Average       16       5       6      72    4.00

Open in new window


# mpstat 1 1
System configuration: lcpu=8 mode=Capped 

cpu  min  maj  mpc  int   cs  ics   rq  mig lpa sysc us sy wt id   pc
  0 4287   21    0 1152 1248  692    0  136 100 25903 34 14 29 23 0.68
  1    0    1    0  305   22   11    0    0 100  124  0  2  5 93 0.31
  2 1719   18    0 1342 1671  861    0  144 100 10558 23  8 27 42 0.61
  3    3    0    0  339   45   19    0    7 100   61  0  1  0 99 0.39
  4 3747    0    0 1047 1248  685    0   90  86 23355 14 13  1 72 0.60
  5    0    0    0  585  296  129    0    2  99  632  0  2  0 98 0.40
  6 2634    0    0 1014 1410  770    0  109  85 11946  8  9  0 82 0.50
  7    0    0    0  538    3    1    0    0 100 1302 27  3  0 69 0.51
ALL 12390   40    0 6322 5943 3168    0  488  94 73881 16  8 10 67 3.99

Open in new window


iostat 10 1
System configuration: lcpu=8 drives=43 paths=42 vdisks=0

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait
          0.9        1732.7               13.6   4.0   79.6      2.8

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk14          0.6       2.4       0.6         24         0
hdisk15          0.0       0.0       0.0          0         0
hdisk39          0.0       0.0       0.0          0         0
hdisk13          0.0       0.0       0.0          0         0
hdisk41          0.0       0.0       0.0          0         0
hdisk37          0.0       0.0       0.0          0         0
hdisk11          0.0       0.0       0.0          0         0
hdisk38          0.0       0.0       0.0          0         0
hdisk36          0.0       0.0       0.0          0         0
hdisk10         17.6     206.1      51.5          0      2060
hdisk12         19.1     206.1      51.5          0      2060
hdisk40          0.0       0.0       0.0          0         0
hdisk17          7.2      73.4      18.2          0       734
hdisk19         11.6     125.4      28.5          0      1254
hdisk16         10.9     128.4      23.6          0      1284
hdisk18         11.9     145.4      26.0          0      1453
hdisk21         10.4      98.6      23.9          0       986
hdisk23          0.0       0.0       0.0          0         0
hdisk22          0.0       0.0       0.0          0         0
hdisk27         10.5     100.2      23.9         16       986
hdisk25         11.5     125.4      28.5          0      1254
hdisk29          8.4     140.6      19.6          0      1405
hdisk24         11.5     145.4      25.9          0      1453
hdisk35          0.0       0.0       0.0          0         0
hdisk26          8.8      73.4      18.2          0       734
hdisk20          8.3     140.6      19.6          0      1405
hdisk28         12.5     128.4      23.6          0      1284
hdisk31          3.5      57.3       8.4          0       573
hdisk4          12.0     145.4      26.0          0      1453
hdisk1          10.3     128.4      23.6          0      1284
hdisk0          10.1      98.6      23.9          0       986
hdisk2           8.0      73.4      18.2          0       734
hdisk5           7.8     140.6      19.6          0      1405
hdisk33          0.0       0.0       0.0          0         0
hdisk34          0.0       0.0       0.0          0         0
hdisk32          0.0       0.0       0.0          0         0
hdisk3          12.4     125.4      28.5          0      1254
hdisk7           3.7      58.1       8.5          8       573
hdisk6           4.7      98.3      10.3          0       983
hdisk30          4.5      99.1      10.4          8       983
hdisk9           0.0       0.0       0.0          0         0
hdisk8           0.0       0.0       0.0          0         0
cd0              0.0       0.0       0.0          0         0

Open in new window


What I am trying to do is ensure system is not overloaded, and that buffer memory is not trashing the disk because our OLTP database would grind to a crawl in such cases.

Any hints or techniques are appreciated :)

If you require more output from different commands, please post and I will post results.
0
Comment
Question by:mirde
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 2

Expert Comment

by:ExpertSteve
ID: 34142653
Well, from that short set of data, it was not overloaded during the times you captured that output.
Your % iowait is low, and no disk is active much of the time. You have no swapping, and plenty of idle CPU. (Which, given it is not blocked waiting for IO, means you have lots of headroom.)

But you really need to trend these things over time, not just with point in time snapshots.
Checkout something like http://www.logicmonitor.com/monitoring/servers/aix/
0
 
LVL 62

Accepted Solution

by:
gheist earned 250 total points
ID: 34144578
huge wa in vmstat comes from active swapping.
configure your processes to use less memory and set vmo thresholds to page in bigger chunks.
0
 
LVL 62

Assisted Solution

by:gheist
gheist earned 250 total points
ID: 34211390
wa is IO wait.
i'd blindly suggest reorgvg...
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
This article provides a convenient collection of links to Microsoft provided Security Patches for operating systems that have reached their End of Life support cycle. Included operating systems covered by this article are Windows XP,  Windows Server…
This tutorial will walk an individual through the steps necessary to configure their installation of BackupExec 2012 to use network shared disk space. Verify that the path to the shared storage is valid and that data can be written to that location:…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

617 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question