Iowait is very high help to find issue and solution for the same
Here are some new outputs :
Please help me to find the issue
according to me the problem is in dm-8 disk but how i can confirm it and also if that disk is a problem how to troubleshoot or resolve it:
00:00:01 CPU %user %nice %system %iowait %steal %idle
00:10:01 all 0.51 0.09 0.77 12.31 0.00 86.31
00:10:01 0 1.86 0.59 3.54 83.91 0.00 10.11
00:10:01 1 0.21 0.00 0.44 0.03 0.00 99.32
00:10:01 2 0.74 0.04 0.60 5.07 0.00 93.56
00:10:01 3 0.35 0.00 0.65 0.32 0.00 98.68
00:10:01 4 0.15 0.04 0.25 4.84 0.00 94.71
00:10:01 5 0.13 0.00 0.25 0.07 0.00 99.55
00:10:01 6 0.12 0.07 0.22 4.17 0.00 95.41
00:10:01 7 0.55 0.00 0.19 0.10 0.00 99.16
00:20:01 all 0.53 0.07 0.78 12.27 0.00 86.34
00:20:01 0 1.97 0.48 3.74 84.60 0.00 9.20
00:20:01 1 0.23 0.00 0.41 0.04 0.00 99.33
00:20:01 2 0.72 0.02 0.58 4.06 0.00 94.62
00:20:01 3 0.36 0.00 0.63 0.37 0.00 98.63
00:20:01 4 0.17 0.01 0.27 5.74 0.00 93.80
00:20:01 5 0.14 0.00 0.25 0.10 0.00 99.51
00:20:01 6 0.15 0.02 0.17 3.20 0.00 96.45
00:20:01 7 0.53 0.00 0.18 0.07 0.00 99.21
00:30:01 all 0.49 0.07 0.74 12.22 0.00 86.49
00:30:01 0 1.79 0.45 3.49 85.07 0.00 9.21
00:30:01 1 0.21 0.00 0.40 0.03 0.00 99.36
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
cciss/c0d0 221.68 2746.92 1629.82 52216455383 30981429987
cciss/c0d0p1 0.03 0.13 0.00 2555493 7095
cciss/c0d0p2 221.65 2746.77 1629.82 52213517896 30981422892
cciss/c0d1 0.17 1.37 0.00 26082406 0
cciss/c0d1p1 0.17 1.35 0.00 25709932 0
sda 0.00 0.00 0.00 53584 0
sdb 1.46 43.08 6.50 818881148 123644544
sdb3 1.46 43.08 6.50 818879204 123644136
sdc 7.44 837.66 786.11 15923240363 14943185456
sdc3 7.44 837.66 786.11 15923238139 14943185088
sdd 0.00 0.01 0.00 120664 0
sdd3 0.00 0.01 0.00 118712 0
sde 0.05 0.57 0.05 10761426 945167
sde3 0.05 0.57 0.05 10759066 944967
sdf 0.00 0.01 0.00 114727 0
sdf3 0.00 0.01 0.00 112703 0
sdg 0.00 0.01 0.00 115770 0
sdg3 0.00 0.01 0.00 113450 0
sdh 0.00 0.01 0.00 115519 0
sdh3 0.00 0.01 0.00 113231 0
sdi 0.00 0.01 0.00 115873 0
sdi3 0.00 0.01 0.00 113497 0
sdj 0.00 0.01 0.00 122730 0
sdj3 0.00 0.01 0.00 120210 0
sdk 0.97 37.82 5.64 719013214 107230736
sdk3 0.97 37.82 5.64 719010286 107230360
sdl 11.09 1522.57 1687.99 28942605873 32087113426
sdl3 11.09 1522.57 1687.99 28942602905 32087112994
sdm 0.62 44.29 1.93 841904229 36733390
sdm3 0.62 44.29 1.93 841901333 36733046
sdn 1.02 40.31 3.12 766250554 59276673
sdn3 1.02 40.31 3.12 766247386 59276289
sdo 0.34 21.04 3.79 399967216 72003498
sdo3 0.34 21.04 3.79 399964064 72003090
sdp 0.00 0.00 0.00 61464 0
sdq 1.46 43.06 6.51 818562686 123670001
sdq3 1.46 43.06 6.51 818559606 123669641
sdr 7.44 837.43 786.59 15918832977 14952317741
sdr3 7.44 837.43 786.59 15918829865 14952317341
sds 0.00 0.01 0.00 122979 0
sds3 0.00 0.01 0.00 120267 0
sdt 0.05 0.56 0.05 10716524 929714
sdt3 0.05 0.56 0.05 10713500 929466
sdu 0.00 0.01 0.00 118253 0
sdu3 0.00 0.01 0.00 115469 0
sdv 0.00 0.01 0.00 117866 0
sdv3 0.00 0.01 0.00 115146 0
sdw 0.00 0.01 0.00 117141 0
sdw3 0.00 0.01 0.00 114613 0
sdx 0.00 0.01 0.00 117643 0
sdx3 0.00 0.01 0.00 115059 0
sdy 0.00 0.01 0.00 123217 0
sdy3 0.00 0.01 0.00 120561 0
sdz 0.97 37.81 5.64 718696076 107253497
sdz3 0.97 37.81 5.64 718692948 107253105
sdaa 11.10 1522.39 1688.04 28939283700 32088033224
sdaa3 11.10 1522.39 1688.04 28939280436 32088032888
sdab 0.62 44.28 1.94 841699239 36806084
sdab3 0.62 44.28 1.94 841696127 36805660
sdac 1.02 40.35 3.11 767016166 59197545
sdac3 1.02 40.35 3.11 767013278 59197161
sdad 0.34 21.08 3.79 400651889 72072365
sdad3 0.34 21.08 3.79 400648913 72072005
sdae 0.00 0.00 0.00 58544 0
sdaf 0.00 0.01 0.00 123409 0
sdaf3 0.00 0.01 0.00 120841 0
sdag 0.00 0.01 0.00 124419 0
sdag3 0.00 0.01 0.00 121659 0
sdah 0.00 0.01 0.00 121907 0
sdah3 0.00 0.01 0.00 119627 0
sdai 0.86 0.01 29.97 230476 569639003
sdai3 0.86 0.01 29.97 227588 569638643
sdaj 0.33 0.01 6.55 185038 124600858
sdaj3 0.33 0.01 6.55 181966 124600458
sdak 0.00 0.01 0.05 250679 937159
sdak3 0.00 0.01 0.05 247615 936935
sdal 0.00 0.01 0.00 118096 0
sdal3 0.00 0.01 0.00 115376 0
sdam 0.00 0.01 0.00 118636 0
sdam3 0.00 0.01 0.00 115916 0
sdan 0.00 0.01 0.00 116216 0
sdan3 0.00 0.01 0.00 114016 0
sdao 0.00 0.01 0.00 115971 0
sdao3 0.00 0.01 0.00 113851 0
sdap 0.06 0.01 1.92 132184 36504143
sdap3 0.06 0.01 1.92 129792 36503759
sdaq 0.18 0.01 3.09 102368 58709665
sdaq3 0.18 0.01 3.09 99920 58709257
sdar 0.32 0.01 6.30 105405 119721467
sdar3 0.32 0.01 6.30 102541 119721123
sdas 8.21 0.01 2445.16 107164 46480336672
sdas3 8.21 0.01 2445.16 104636 46480336216
sdat 0.15 0.01 5.84 247300 110972035
sdat3 0.15 0.01 5.84 244396 110971619
sdau 0.00 0.00 0.00 61136 0
sdav 0.00 0.01 0.00 122711 0
sdav3 0.00 0.01 0.00 120351 0
sdaw 0.00 0.01 0.00 122837 0
sdaw3 0.00 0.01 0.00 120405 0
sdax 0.00 0.01 0.00 122504 0
sdax3 0.00 0.01 0.00 120128 0
sday 0.86 0.01 29.99 233031 570097679
sday3 0.86 0.01 29.99 230247 570097271
sdaz 0.33 0.01 6.57 177183 124800653
sdaz3 0.33 0.01 6.57 174407 124800285
sdba 0.00 0.01 0.05 261409 927064
sdba3 0.00 0.01 0.05 258801 926840
sdbb 0.00 0.01 0.00 117044 0
sdbb3 0.00 0.01 0.00 114676 0
sdbc 0.00 0.01 0.00 116764 0
sdbc3 0.00 0.01 0.00 114508 0
sdbd 0.00 0.01 0.00 116396 0
sdbd3 0.00 0.01 0.00 114164 0
sdbe 0.00 0.01 0.00 115729 0
sdbe3 0.00 0.01 0.00 113577 0
sdbf 0.06 0.01 1.92 126283 36457252
sdbf3 0.06 0.01 1.92 123739 36456868
sdbg 0.18 0.01 3.09 104413 58711640
sdbg3 0.18 0.01 3.09 101493 58711280
sdbh 0.32 0.01 6.31 104300 119859587
sdbh3 0.32 0.01 6.31 101708 119859163
sdbi 8.22 0.01 2444.56 103377 46468918356
sdbi3 8.22 0.01 2444.56 100705 46468918044
sdbj 0.15 0.01 5.84 264340 110937704
sdbj3 0.15 0.01 5.84 261812 110937352
dm-0 10.44 170.04 37.26 3232313330 708204240
dm-1 0.00 0.01 0.00 134038 10012
dm-2 0.00 0.01 0.01 214066 232800
dm-3 75.57 180.42 434.66 3429712690 8262499160
dm-4 0.99 0.86 7.67 16404890 145791440
dm-5 0.03 3.43 0.01 65173682 128080
dm-6 3.59 37.91 18.32 720633490 348272208
dm-7 0.00 0.02 0.00 286892 21888
dm-8 364.88 1840.09 1078.99 34978455192 20510597224
dm-9 16.25 508.44 52.80 9664969778 1003771880
dm-10 0.08 4.81 0.10 91352570 1893960
sdbk 0.72 16.27 5.33 309367822 101314508
sdbk3 0.72 16.27 5.33 309362616 101314107
sdbl 0.33 0.01 10.80 120116 205216840
sdbl3 0.33 0.01 10.80 115111 205216431
sdbm 0.72 16.30 5.33 309780980 101338853
sdbm3 0.72 16.30 5.33 309775871 101338436
sdbn 0.33 0.01 10.80 130042 205247535
sdbn3 0.33 0.01 10.80 125096 205247134
sdbo 0.00 0.00 0.00 21408 0
sdbp 0.00 0.00 0.00 22232 0
sdbq 0.00 3.72 0.01 70757640 191495
sdbq3 0.00 3.72 0.01 70754558 191398
sdbr 0.00 3.70 0.01 70425420 242391
sdbr3 0.00 3.70 0.01 70421638 242270
sdbs 0.00 0.00 3.73 49506 70854350
sdbs3 0.00 0.00 3.73 46199 70854269
sdbt 0.00 0.00 3.71 51963 70598236
sdbt3 0.00 0.00 3.71 49406 70598099
VxVM18000 2.85 137.05 9.67 2605252785 183827904
VxVM4000 1.30 19.43 5.81 369312078 110479044
VxVM4001 0.08 10.03 1.15 190580192 21858248
VxVM4003 0.02 2.12 0.04 40371240 807645
VxVM4002 4.47 828.63 824.38 15751424152 15670738824
VxVM3000 0.02 0.16 0.01 2960772 277267
Please help me to find the issue
according to me the problem is in dm-8 disk but how i can confirm it and also if that disk is a problem how to troubleshoot or resolve it:
00:00:01 CPU %user %nice %system %iowait %steal %idle
00:10:01 all 0.51 0.09 0.77 12.31 0.00 86.31
00:10:01 0 1.86 0.59 3.54 83.91 0.00 10.11
00:10:01 1 0.21 0.00 0.44 0.03 0.00 99.32
00:10:01 2 0.74 0.04 0.60 5.07 0.00 93.56
00:10:01 3 0.35 0.00 0.65 0.32 0.00 98.68
00:10:01 4 0.15 0.04 0.25 4.84 0.00 94.71
00:10:01 5 0.13 0.00 0.25 0.07 0.00 99.55
00:10:01 6 0.12 0.07 0.22 4.17 0.00 95.41
00:10:01 7 0.55 0.00 0.19 0.10 0.00 99.16
00:20:01 all 0.53 0.07 0.78 12.27 0.00 86.34
00:20:01 0 1.97 0.48 3.74 84.60 0.00 9.20
00:20:01 1 0.23 0.00 0.41 0.04 0.00 99.33
00:20:01 2 0.72 0.02 0.58 4.06 0.00 94.62
00:20:01 3 0.36 0.00 0.63 0.37 0.00 98.63
00:20:01 4 0.17 0.01 0.27 5.74 0.00 93.80
00:20:01 5 0.14 0.00 0.25 0.10 0.00 99.51
00:20:01 6 0.15 0.02 0.17 3.20 0.00 96.45
00:20:01 7 0.53 0.00 0.18 0.07 0.00 99.21
00:30:01 all 0.49 0.07 0.74 12.22 0.00 86.49
00:30:01 0 1.79 0.45 3.49 85.07 0.00 9.21
00:30:01 1 0.21 0.00 0.40 0.03 0.00 99.36
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
cciss/c0d0 221.68 2746.92 1629.82 52216455383 30981429987
cciss/c0d0p1 0.03 0.13 0.00 2555493 7095
cciss/c0d0p2 221.65 2746.77 1629.82 52213517896 30981422892
cciss/c0d1 0.17 1.37 0.00 26082406 0
cciss/c0d1p1 0.17 1.35 0.00 25709932 0
sda 0.00 0.00 0.00 53584 0
sdb 1.46 43.08 6.50 818881148 123644544
sdb3 1.46 43.08 6.50 818879204 123644136
sdc 7.44 837.66 786.11 15923240363 14943185456
sdc3 7.44 837.66 786.11 15923238139 14943185088
sdd 0.00 0.01 0.00 120664 0
sdd3 0.00 0.01 0.00 118712 0
sde 0.05 0.57 0.05 10761426 945167
sde3 0.05 0.57 0.05 10759066 944967
sdf 0.00 0.01 0.00 114727 0
sdf3 0.00 0.01 0.00 112703 0
sdg 0.00 0.01 0.00 115770 0
sdg3 0.00 0.01 0.00 113450 0
sdh 0.00 0.01 0.00 115519 0
sdh3 0.00 0.01 0.00 113231 0
sdi 0.00 0.01 0.00 115873 0
sdi3 0.00 0.01 0.00 113497 0
sdj 0.00 0.01 0.00 122730 0
sdj3 0.00 0.01 0.00 120210 0
sdk 0.97 37.82 5.64 719013214 107230736
sdk3 0.97 37.82 5.64 719010286 107230360
sdl 11.09 1522.57 1687.99 28942605873 32087113426
sdl3 11.09 1522.57 1687.99 28942602905 32087112994
sdm 0.62 44.29 1.93 841904229 36733390
sdm3 0.62 44.29 1.93 841901333 36733046
sdn 1.02 40.31 3.12 766250554 59276673
sdn3 1.02 40.31 3.12 766247386 59276289
sdo 0.34 21.04 3.79 399967216 72003498
sdo3 0.34 21.04 3.79 399964064 72003090
sdp 0.00 0.00 0.00 61464 0
sdq 1.46 43.06 6.51 818562686 123670001
sdq3 1.46 43.06 6.51 818559606 123669641
sdr 7.44 837.43 786.59 15918832977 14952317741
sdr3 7.44 837.43 786.59 15918829865 14952317341
sds 0.00 0.01 0.00 122979 0
sds3 0.00 0.01 0.00 120267 0
sdt 0.05 0.56 0.05 10716524 929714
sdt3 0.05 0.56 0.05 10713500 929466
sdu 0.00 0.01 0.00 118253 0
sdu3 0.00 0.01 0.00 115469 0
sdv 0.00 0.01 0.00 117866 0
sdv3 0.00 0.01 0.00 115146 0
sdw 0.00 0.01 0.00 117141 0
sdw3 0.00 0.01 0.00 114613 0
sdx 0.00 0.01 0.00 117643 0
sdx3 0.00 0.01 0.00 115059 0
sdy 0.00 0.01 0.00 123217 0
sdy3 0.00 0.01 0.00 120561 0
sdz 0.97 37.81 5.64 718696076 107253497
sdz3 0.97 37.81 5.64 718692948 107253105
sdaa 11.10 1522.39 1688.04 28939283700 32088033224
sdaa3 11.10 1522.39 1688.04 28939280436 32088032888
sdab 0.62 44.28 1.94 841699239 36806084
sdab3 0.62 44.28 1.94 841696127 36805660
sdac 1.02 40.35 3.11 767016166 59197545
sdac3 1.02 40.35 3.11 767013278 59197161
sdad 0.34 21.08 3.79 400651889 72072365
sdad3 0.34 21.08 3.79 400648913 72072005
sdae 0.00 0.00 0.00 58544 0
sdaf 0.00 0.01 0.00 123409 0
sdaf3 0.00 0.01 0.00 120841 0
sdag 0.00 0.01 0.00 124419 0
sdag3 0.00 0.01 0.00 121659 0
sdah 0.00 0.01 0.00 121907 0
sdah3 0.00 0.01 0.00 119627 0
sdai 0.86 0.01 29.97 230476 569639003
sdai3 0.86 0.01 29.97 227588 569638643
sdaj 0.33 0.01 6.55 185038 124600858
sdaj3 0.33 0.01 6.55 181966 124600458
sdak 0.00 0.01 0.05 250679 937159
sdak3 0.00 0.01 0.05 247615 936935
sdal 0.00 0.01 0.00 118096 0
sdal3 0.00 0.01 0.00 115376 0
sdam 0.00 0.01 0.00 118636 0
sdam3 0.00 0.01 0.00 115916 0
sdan 0.00 0.01 0.00 116216 0
sdan3 0.00 0.01 0.00 114016 0
sdao 0.00 0.01 0.00 115971 0
sdao3 0.00 0.01 0.00 113851 0
sdap 0.06 0.01 1.92 132184 36504143
sdap3 0.06 0.01 1.92 129792 36503759
sdaq 0.18 0.01 3.09 102368 58709665
sdaq3 0.18 0.01 3.09 99920 58709257
sdar 0.32 0.01 6.30 105405 119721467
sdar3 0.32 0.01 6.30 102541 119721123
sdas 8.21 0.01 2445.16 107164 46480336672
sdas3 8.21 0.01 2445.16 104636 46480336216
sdat 0.15 0.01 5.84 247300 110972035
sdat3 0.15 0.01 5.84 244396 110971619
sdau 0.00 0.00 0.00 61136 0
sdav 0.00 0.01 0.00 122711 0
sdav3 0.00 0.01 0.00 120351 0
sdaw 0.00 0.01 0.00 122837 0
sdaw3 0.00 0.01 0.00 120405 0
sdax 0.00 0.01 0.00 122504 0
sdax3 0.00 0.01 0.00 120128 0
sday 0.86 0.01 29.99 233031 570097679
sday3 0.86 0.01 29.99 230247 570097271
sdaz 0.33 0.01 6.57 177183 124800653
sdaz3 0.33 0.01 6.57 174407 124800285
sdba 0.00 0.01 0.05 261409 927064
sdba3 0.00 0.01 0.05 258801 926840
sdbb 0.00 0.01 0.00 117044 0
sdbb3 0.00 0.01 0.00 114676 0
sdbc 0.00 0.01 0.00 116764 0
sdbc3 0.00 0.01 0.00 114508 0
sdbd 0.00 0.01 0.00 116396 0
sdbd3 0.00 0.01 0.00 114164 0
sdbe 0.00 0.01 0.00 115729 0
sdbe3 0.00 0.01 0.00 113577 0
sdbf 0.06 0.01 1.92 126283 36457252
sdbf3 0.06 0.01 1.92 123739 36456868
sdbg 0.18 0.01 3.09 104413 58711640
sdbg3 0.18 0.01 3.09 101493 58711280
sdbh 0.32 0.01 6.31 104300 119859587
sdbh3 0.32 0.01 6.31 101708 119859163
sdbi 8.22 0.01 2444.56 103377 46468918356
sdbi3 8.22 0.01 2444.56 100705 46468918044
sdbj 0.15 0.01 5.84 264340 110937704
sdbj3 0.15 0.01 5.84 261812 110937352
dm-0 10.44 170.04 37.26 3232313330 708204240
dm-1 0.00 0.01 0.00 134038 10012
dm-2 0.00 0.01 0.01 214066 232800
dm-3 75.57 180.42 434.66 3429712690 8262499160
dm-4 0.99 0.86 7.67 16404890 145791440
dm-5 0.03 3.43 0.01 65173682 128080
dm-6 3.59 37.91 18.32 720633490 348272208
dm-7 0.00 0.02 0.00 286892 21888
dm-8 364.88 1840.09 1078.99 34978455192 20510597224
dm-9 16.25 508.44 52.80 9664969778 1003771880
dm-10 0.08 4.81 0.10 91352570 1893960
sdbk 0.72 16.27 5.33 309367822 101314508
sdbk3 0.72 16.27 5.33 309362616 101314107
sdbl 0.33 0.01 10.80 120116 205216840
sdbl3 0.33 0.01 10.80 115111 205216431
sdbm 0.72 16.30 5.33 309780980 101338853
sdbm3 0.72 16.30 5.33 309775871 101338436
sdbn 0.33 0.01 10.80 130042 205247535
sdbn3 0.33 0.01 10.80 125096 205247134
sdbo 0.00 0.00 0.00 21408 0
sdbp 0.00 0.00 0.00 22232 0
sdbq 0.00 3.72 0.01 70757640 191495
sdbq3 0.00 3.72 0.01 70754558 191398
sdbr 0.00 3.70 0.01 70425420 242391
sdbr3 0.00 3.70 0.01 70421638 242270
sdbs 0.00 0.00 3.73 49506 70854350
sdbs3 0.00 0.00 3.73 46199 70854269
sdbt 0.00 0.00 3.71 51963 70598236
sdbt3 0.00 0.00 3.71 49406 70598099
VxVM18000 2.85 137.05 9.67 2605252785 183827904
VxVM4000 1.30 19.43 5.81 369312078 110479044
VxVM4001 0.08 10.03 1.15 190580192 21858248
VxVM4003 0.02 2.12 0.04 40371240 807645
VxVM4002 4.47 828.63 824.38 15751424152 15670738824
VxVM3000 0.02 0.16 0.01 2960772 277267
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Have you been able to verify that dm-8 is your paging device and that the paging rate is indeed high?
To find the top memory-consuming processes start "top" and check the "VIRT" column.
To find the top memory-consuming processes start "top" and check the "VIRT" column.
ASKER
If I top than Sendmail ,vmxciad ,Patrol agent ,perl ,cmafcd etc all showing high in VIRT and Vmstat shows:
procs -----------memory---------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 3538776 3089640 89808 291256 115 68 491 408 0 0 1 1 92 6 0
m not able to find the details about dm-8 :(
procs -----------memory---------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 3538776 3089640 89808 291256 115 68 491 408 0 0 1 1 92 6 0
m not able to find the details about dm-8 :(
Where is your swap space:
/etc/fstab # (second column = "swap")
swapon -s
cat /proc/swaps
Map LVM logical devices:
lvdisplay|awk '/LV Name/{n=$3} /Block device/{d=$3; sub(".*:","dm-",d); print d,n;}'
Look for dm-8.
/etc/fstab # (second column = "swap")
swapon -s
cat /proc/swaps
Map LVM logical devices:
lvdisplay|awk '/LV Name/{n=$3} /Block device/{d=$3; sub(".*:","dm-",d); print d,n;}'
Look for dm-8.
dmsetup info /dev/dm-8
should do the trick.
It can be an LVM volume, dm-raid set
should do the trick.
It can be an LVM volume, dm-raid set
ASKER
yes dm-8 is a swap disk
dm-8 /dev/rootvg/lv_swap
cat /proc/swaps
Filename Type Size Used Priority
/dev/mapper/rootvg-lv_swap
Now what should i increase it or what ?
dm-8 /dev/rootvg/lv_swap
cat /proc/swaps
Filename Type Size Used Priority
/dev/mapper/rootvg-lv_swap
Now what should i increase it or what ?
You should examine the output of "top" (the "VIRT" column, remember?) for processes consuming excessive memory.
If there are no such processes or if you can't change their behaviour in regard to memory consumption you will have to reduce the load of that server (move some applications to another machine, for example) or buy more memory.
If there are no such processes or if you can't change their behaviour in regard to memory consumption you will have to reduce the load of that server (move some applications to another machine, for example) or buy more memory.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
if that disk is a regular sata, it seems to be used at the maximum of it's capacities. the limitation is connected to the number of transactions as such disks usually cannot give a decent throughput once you reached 250-400 tps (the limit varies depending on the disk and OS)
what actually is your problem ? the OS run ? the specs of the machine ? it's use ? your goal ?
what actually is your problem ? the OS run ? the specs of the machine ? it's use ? your goal ?
ASKER
Problem is with IOWAIT High and CPU high usage
OS - RHEL5.2
2.6.18-128.1.14.el5 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
Firmware Revision: 1.82
Handle 0x0100, DMI type 1, 27 bytes.
System Information
Manufacturer: HP
Product Name: ProLiant DL380 G6
Version: Not Specified
Serial Number: XXXXXXXX
UUID: XXXXXXXXXXXXXXX
Wake-up Type: Power Switch
SKU Number: XXXXXXXXX
Family: ProLiant
Handle 0x0300, DMI type 3, 21 bytes.
Chassis Information
Manufacturer: HP
Type: Rack Mount Chassis
Lock: Not Present
Version: Not Specified
Serial Number: XXXXXXXX
Asset Tag:
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Heigth: 2 U
Some recent strange logs:
AutonomyContent[22550]: segfault at 000000000000000a rip 0000000000ba93d6 rsp 00000000e52d8010 error 4
tcp 0 0 0.0.0.0:9024 0.0.0.0:* LISTEN 16833/AutonomyConte
tcp 0 0 0.0.0.0:9025 0.0.0.0:* LISTEN 16833/AutonomyConte
tcp 0 0 0.0.0.0:9026 0.0.0.0:* LISTEN 16833/AutonomyConte
tcp 0 0 0.0.0.0:9027 0.0.0.0:* LISTEN 16836/AutonomyConte
tcp 0 0 0.0.0.0:9028 0.0.0.0:* LISTEN 16836/AutonomyConte
tcp 0 0 0.0.0.0:9029 0.0.0.0:* LISTEN 16836/AutonomyConte
tcp 0 0 0.0.0.0:9030 0.0.0.0:* LISTEN 16839/AutonomyConte
tcp 0 0 0.0.0.0:9031 0.0.0.0:* LISTEN 16839/AutonomyConte
tcp 0 0 0.0.0.0:9032 0.0.0.0:* LISTEN 16839/AutonomyConte
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 16806/AutonomyDiSH.
tcp 0 0 0.0.0.0:9033 0.0.0.0:* LISTEN 16842/AutonomyConte
tcp 0 0 0.0.0.0:9001 0.0.0.0:* LISTEN 16806/AutonomyDiSH.
tcp 0 0 0.0.0.0:9034 0.0.0.0:* LISTEN 16842/AutonomyConte
tcp 0 0 0.0.0.0:9035 0.0.0.0:* LISTEN 16842/AutonomyConte
tcp 0 0 0.0.0.0:9036 0.0.0.0:* LISTEN 16845/AutonomyConte
tcp 0 0 0.0.0.0:9037 0.0.0.0:* LISTEN 16845/AutonomyConte
tcp 0 0 0.0.0.0:9038 0.0.0.0:* LISTEN 16845/AutonomyConte
tcp 0 0 0.0.0.0:9039 0.0.0.0:* LISTEN 16848/AutonomyConte
root 3503 0.0 0.0 61176 728 pts/1 S+ 14:34 0:00 grep Autonomy
hpwd 16806 0.0 0.0 75676 2584 ? Sl Jul10 0:14 ./AutonomyDiSH.exe
hpwd 16833 0.0 2.0 468436 170636 ? Sl Jul10 79:47 ./AutonomyContentServer.ex
hpwd 16836 0.0 2.0 471632 171384 ? Sl Jul10 89:48 ./AutonomyContentServer.ex
hpwd 16839 0.0 2.0 473268 169420 ? Sl Jul10 86:13 ./AutonomyContentServer.ex
hpwd 16842 0.0 2.0 472536 171088 ? Sl Jul10 85:01 ./AutonomyContentServer.ex
hpwd 16845 0.0 2.1 471912 172940 ? Sl Jul10 88:21 ./AutonomyContentServer.ex
hpwd 16848 0.0 2.0 470200 168692 ? Sl Jul10 81:24 ./AutonomyContentServer.ex
hpwd 16854 0.0 2.1 472120 171764 ? Sl Jul10 83:28 ./AutonomyContentServer.ex
hpwd 16857 0.3 1.2 211204 99924 ? Sl Jul10 357:11 ./AutonomyFileSystemFetch.
hpwd 16860 0.3 1.0 196208 85636 ? Sl Jul10 353:08 ./AutonomyFileSystemFetch.
hpwd 16864 0.3 1.0 197456 86956 ? Sl Jul10 356:53 ./AutonomyFileSystemFetch.
hpwd 16867 0.3 1.2 210248 98584 ? Sl Jul10 361:21 ./AutonomyFileSystemFetch.
hpwd 16870 0.3 1.0 203684 87376 ? Sl Jul10 357:23 ./AutonomyFileSystemFetch.
hpwd 16873 0.3 1.2 209444 98332 ? Sl Jul10 359:08 ./AutonomyFileSystemFetch.
hpwd 16876 0.3 1.0 209912 88876 ? Sl Jul10 358:07 ./AutonomyFileSystemFetch.
hpwd 16879 0.3 1.0 194000 84488 ? Sl Jul10 363:16 ./AutonomyFileSystemFetch.
############### TOP 10 Memory Consuming ###################
1.38 GB 13003 hpwd /sites/HPWD/hpwd/mysql/bin
812.56 MB 16885 hpwd ./dah.exe
420.43 MB 16839 hpwd ./AutonomyContentServer.ex
419.71 MB 16842 hpwd ./AutonomyContentServer.ex
419.30 MB 16854 hpwd ./AutonomyContentServer.ex
419.10 MB 16845 hpwd ./AutonomyContentServer.ex
418.83 MB 16836 hpwd ./AutonomyContentServer.ex
417.43 MB 16848 hpwd ./AutonomyContentServer.ex
415.71 MB 16833 hpwd ./AutonomyContentServer.ex
OS - RHEL5.2
2.6.18-128.1.14.el5 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
Firmware Revision: 1.82
Handle 0x0100, DMI type 1, 27 bytes.
System Information
Manufacturer: HP
Product Name: ProLiant DL380 G6
Version: Not Specified
Serial Number: XXXXXXXX
UUID: XXXXXXXXXXXXXXX
Wake-up Type: Power Switch
SKU Number: XXXXXXXXX
Family: ProLiant
Handle 0x0300, DMI type 3, 21 bytes.
Chassis Information
Manufacturer: HP
Type: Rack Mount Chassis
Lock: Not Present
Version: Not Specified
Serial Number: XXXXXXXX
Asset Tag:
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Heigth: 2 U
Some recent strange logs:
AutonomyContent[22550]: segfault at 000000000000000a rip 0000000000ba93d6 rsp 00000000e52d8010 error 4
tcp 0 0 0.0.0.0:9024 0.0.0.0:* LISTEN 16833/AutonomyConte
tcp 0 0 0.0.0.0:9025 0.0.0.0:* LISTEN 16833/AutonomyConte
tcp 0 0 0.0.0.0:9026 0.0.0.0:* LISTEN 16833/AutonomyConte
tcp 0 0 0.0.0.0:9027 0.0.0.0:* LISTEN 16836/AutonomyConte
tcp 0 0 0.0.0.0:9028 0.0.0.0:* LISTEN 16836/AutonomyConte
tcp 0 0 0.0.0.0:9029 0.0.0.0:* LISTEN 16836/AutonomyConte
tcp 0 0 0.0.0.0:9030 0.0.0.0:* LISTEN 16839/AutonomyConte
tcp 0 0 0.0.0.0:9031 0.0.0.0:* LISTEN 16839/AutonomyConte
tcp 0 0 0.0.0.0:9032 0.0.0.0:* LISTEN 16839/AutonomyConte
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 16806/AutonomyDiSH.
tcp 0 0 0.0.0.0:9033 0.0.0.0:* LISTEN 16842/AutonomyConte
tcp 0 0 0.0.0.0:9001 0.0.0.0:* LISTEN 16806/AutonomyDiSH.
tcp 0 0 0.0.0.0:9034 0.0.0.0:* LISTEN 16842/AutonomyConte
tcp 0 0 0.0.0.0:9035 0.0.0.0:* LISTEN 16842/AutonomyConte
tcp 0 0 0.0.0.0:9036 0.0.0.0:* LISTEN 16845/AutonomyConte
tcp 0 0 0.0.0.0:9037 0.0.0.0:* LISTEN 16845/AutonomyConte
tcp 0 0 0.0.0.0:9038 0.0.0.0:* LISTEN 16845/AutonomyConte
tcp 0 0 0.0.0.0:9039 0.0.0.0:* LISTEN 16848/AutonomyConte
root 3503 0.0 0.0 61176 728 pts/1 S+ 14:34 0:00 grep Autonomy
hpwd 16806 0.0 0.0 75676 2584 ? Sl Jul10 0:14 ./AutonomyDiSH.exe
hpwd 16833 0.0 2.0 468436 170636 ? Sl Jul10 79:47 ./AutonomyContentServer.ex
hpwd 16836 0.0 2.0 471632 171384 ? Sl Jul10 89:48 ./AutonomyContentServer.ex
hpwd 16839 0.0 2.0 473268 169420 ? Sl Jul10 86:13 ./AutonomyContentServer.ex
hpwd 16842 0.0 2.0 472536 171088 ? Sl Jul10 85:01 ./AutonomyContentServer.ex
hpwd 16845 0.0 2.1 471912 172940 ? Sl Jul10 88:21 ./AutonomyContentServer.ex
hpwd 16848 0.0 2.0 470200 168692 ? Sl Jul10 81:24 ./AutonomyContentServer.ex
hpwd 16854 0.0 2.1 472120 171764 ? Sl Jul10 83:28 ./AutonomyContentServer.ex
hpwd 16857 0.3 1.2 211204 99924 ? Sl Jul10 357:11 ./AutonomyFileSystemFetch.
hpwd 16860 0.3 1.0 196208 85636 ? Sl Jul10 353:08 ./AutonomyFileSystemFetch.
hpwd 16864 0.3 1.0 197456 86956 ? Sl Jul10 356:53 ./AutonomyFileSystemFetch.
hpwd 16867 0.3 1.2 210248 98584 ? Sl Jul10 361:21 ./AutonomyFileSystemFetch.
hpwd 16870 0.3 1.0 203684 87376 ? Sl Jul10 357:23 ./AutonomyFileSystemFetch.
hpwd 16873 0.3 1.2 209444 98332 ? Sl Jul10 359:08 ./AutonomyFileSystemFetch.
hpwd 16876 0.3 1.0 209912 88876 ? Sl Jul10 358:07 ./AutonomyFileSystemFetch.
hpwd 16879 0.3 1.0 194000 84488 ? Sl Jul10 363:16 ./AutonomyFileSystemFetch.
############### TOP 10 Memory Consuming ###################
1.38 GB 13003 hpwd /sites/HPWD/hpwd/mysql/bin
812.56 MB 16885 hpwd ./dah.exe
420.43 MB 16839 hpwd ./AutonomyContentServer.ex
419.71 MB 16842 hpwd ./AutonomyContentServer.ex
419.30 MB 16854 hpwd ./AutonomyContentServer.ex
419.10 MB 16845 hpwd ./AutonomyContentServer.ex
418.83 MB 16836 hpwd ./AutonomyContentServer.ex
417.43 MB 16848 hpwd ./AutonomyContentServer.ex
415.71 MB 16833 hpwd ./AutonomyContentServer.ex
I think you're running too many "AutonomyContentServer" processes in relation to the main memory available on your machine - thus you're running into paging issues.
As I already told you before - either reduce the number of parallel processes, or, if you really need that many, consider purchasing more memory or moving parts of the load to another machine (if at all possible).
As I already told you before - either reduce the number of parallel processes, or, if you really need that many, consider purchasing more memory or moving parts of the load to another machine (if at all possible).
ASKER
Many thanks
ASKER