Link to home
Create AccountLog in
Avatar of melwong
melwong

asked on

What is the cheapest upgrade I can do based on these server load stats

Enclosed is my server's health check peak time. What is the best cheapest upgrade can I do? Upgrade RAM? HD?

I'm looking for affordable incremental upgrades, not adding more servers. Btw, I can't upgrade the CPU

Server Spec
Processor: Quad Core Intel Xeon 3220 2.4GHz
RAM Memory: 4GB
Hard Drive 1: 160GB SATA
Hard Drive 2: 160GB SATA
Operating System: CentOS Linux
Control Panel: cPanel + Web Host Manager
root@server1 [~]# sar -u 2 50
08:29:06 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
08:29:08 AM       all      0.00      0.38      0.25      0.13      0.00     99.24
08:29:10 AM       all      0.00      0.76      0.13      0.00      0.00     99.11
08:29:12 AM       all      0.00      1.01      0.25      0.51      0.00     98.23
08:29:14 AM       all      0.00      0.25      0.25      0.38      0.00     99.12
08:29:16 AM       all      0.00      0.38      0.13      0.00      0.00     99.49
08:29:18 AM       all      0.00      0.13      0.13      0.00      0.00     99.75
08:29:20 AM       all      0.00      0.76      0.13      0.00      0.00     99.12
08:29:22 AM       all      0.00      0.63      0.38      0.13      0.00     98.87
08:29:24 AM       all      0.00      0.13      0.13      0.13      0.00     99.62
08:29:26 AM       all      0.00      1.52      5.30      0.00      0.00     93.18
08:29:28 AM       all      0.13      2.14      8.70      0.00      0.00     89.03
08:29:30 AM       all      0.00      0.38      0.25      0.25      0.00     99.12
08:29:32 AM       all      0.00      3.28      0.63      0.00      0.00     96.09
08:29:34 AM       all      0.00      0.13      0.00      0.38      0.00     99.49
08:29:36 AM       all      0.00      0.76      0.25      0.25      0.00     98.74
08:29:38 AM       all      0.00      0.38      0.00      0.25      0.00     99.37
08:29:40 AM       all      0.00      0.38      0.25      0.00      0.00     99.37
08:29:42 AM       all      0.00      5.57      1.01      1.52      0.00     91.90
08:29:44 AM       all      0.00      0.88      0.25      0.63      0.00     98.23
08:29:46 AM       all      0.00      0.63      0.25      0.88      0.00     98.23
08:29:48 AM       all      0.00      1.64      0.63      0.13      0.00     97.60
08:29:50 AM       all      0.00      0.63      0.51      0.00      0.00     98.86
08:29:52 AM       all      0.00      0.50      0.25      0.00      0.00     99.24
08:29:54 AM       all      0.00      0.88      0.25      0.00      0.00     98.86
08:29:56 AM       all      0.00      0.25      0.13      0.25      0.00     99.37
08:29:58 AM       all      0.00      1.39      0.38      0.00      0.00     98.23
08:30:00 AM       all      0.00      0.13      0.00      0.00      0.00     99.87
08:30:02 AM       all      6.55      3.78     13.85      0.13      0.00     75.69
08:30:04 AM       all      4.64     13.28     61.15      0.13      0.00     20.80
08:30:06 AM       all      1.50     12.36     58.55      0.37      0.00     27.22
08:30:08 AM       all      0.00     23.78     57.95      0.00      0.00     18.27
08:30:10 AM       all      0.00     12.41     57.77      0.00      0.00     29.82
08:30:12 AM       all      0.00      8.01     45.31      0.25      0.00     46.43
08:30:14 AM       all      0.13      6.88     36.67      0.13      0.00     56.20
08:30:16 AM       all      0.00      6.15     33.12      0.00      0.00     60.73
08:30:18 AM       all      0.00      5.01     32.33      0.00      0.00     62.66
08:30:20 AM       all      0.00      6.51     37.92      0.00      0.00     55.57
08:30:22 AM       all      0.00      7.77     32.33      0.00      0.00     59.90
08:30:24 AM       all      0.00      0.63      1.14      0.25      0.00     97.98
08:30:26 AM       all      0.00      1.89      0.25      0.13      0.00     97.73
08:30:28 AM       all      0.00      0.13      0.13      0.00      0.00     99.75
08:30:30 AM       all      0.00      0.13      0.13      0.00      0.00     99.75
08:30:30 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
08:30:32 AM       all      0.00      1.14      0.13      0.00      0.00     98.74
08:30:34 AM       all      0.00      0.38      0.13      0.00      0.00     99.49
08:30:36 AM       all      0.00      0.13      0.00      0.13      0.00     99.75
08:30:38 AM       all      0.00      9.61      9.48      0.00      0.00     80.91
08:30:40 AM       all      0.00      2.52      1.01      0.00      0.00     96.47
08:30:42 AM       all      0.00      0.38      0.25      0.00      0.00     99.37
08:30:44 AM       all      0.00      0.38      0.00      0.00      0.00     99.62
08:30:46 AM       all      0.00      0.63      0.38      0.25      0.00     98.74
Average:          all      0.26      3.01     10.08      0.15      0.00     86.51


root@server1 [~]# free
             total       used       free     shared    buffers     cached
Mem:       4047632    4009752      37880          0     119844    1448052
-/+ buffers/cache:    2441856    1605776
Swap:      4096532      63948    4032584


root@server1 [~]# ps ax -yl --sort:rss | grep httpd
S     0  2179 31378  0  75   0   720 15297 pipe_w pts/3      0:00 grep httpd
S     0 25597 28759  0  82   4 12512 52354 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2152 28759  0  79   4 16328 61324 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2150 28759  0  79   4 16336 61324 semtim ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16524 28759  0  79   4 17924 61529 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14731 28759  0  79   4 17936 61593 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1399 28759  0  79   4 17984 61593 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32444 28759  0  79   4 17984 61529 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14714 28759  0  79   4 17992 61529 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32673 28759  0  79   4 18052 61529 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31732 28759  0  79   4 18764 61529 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13928 28759  0  81   4 18944 61658 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1212 28759  0  79   4 19140 61529 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14069 28759  0  79   4 19908 62068 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2154 28759  0  79   4 20804 61849 semtim ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16320 28759  0  79   4 20812 61793 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2147 28759  0  80   4 21212 62106 648623 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14650 28759  0  79   4 21364 62226 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2140 28759  0  80   4 21432 62170 117974 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15208 28759  0  79   4 21456 62108 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2159 28759  0  80   4 22376 62202 138805 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32464 28759  0  79   4 22376 62202 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14719 28759  0  79   4 22392 62204 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2149 28759  0  80   4 22416 62700 637110 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16618 28759  0  79   4 22640 62386 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31064 28759  0  80   4 22640 62386 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   731 28759  0  79   4 22716 62425 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1495 28759  0  79   4 22792 62429 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31733 28759  0  79   4 22848 62426 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14739 28759  0  79   4 22864 62295 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 17076 28759  0  79   4 22948 62650 flock_ ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1696 28759  0  79   4 23036 62205 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1498 28759  0  80   4 23184 63269 126474 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S     0 28759     1  0  81   4 23212 61324 -      ?          0:06 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14746 28759  0  79   4 23248 62425 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32462 28759  0  79   4 23440 62446 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   730 28759  0  79   4 23444 62577 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14715 28759  0  79   4 23724 62549 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13991 28759  0  79   4 23748 62486 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31704 28759  0  79   4 23752 62496 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13997 28759  0  79   4 23788 62955 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1438 28759  0  79   4 24260 62856 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14473 28759  0  79   4 24308 63057 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1497 28759  0  80   4 24532 62856 144087 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15459 28759  0  79   4 24580 63044 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   727 28759  0  79   4 24656 63553 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1492 28759  0  80   4 24864 63067 637108 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13998 28759  0  79   4 24928 63136 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16324 28759  0  79   4 24988 63157 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14466 28759  0  79   4 25124 63110 flock_ ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31434 28759  0  79   4 25164 63245 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32416 28759  0  79   4 25224 63154 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14737 28759  0  79   4 25540 63435 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1487 28759  0  79   4 25600 62868 semtim ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1499 28759  0  80   4 25612 63286 112861 ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1472 28759  4  80   4 25616 63356 209092 ?          0:07 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31674 28759  0  79   4 25624 63467 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31089 28759  0  79   4 25724 63332 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1215 28759  0  79   4 25844 63499 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1434 28759  0  79   4 25844 63499 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16322 28759  0  79   4 25884 63529 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15455 28759  0  79   4 25960 63396 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31204 28759  0  79   4 25972 63436 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2148 28759  0  79   4 25988 63358 semtim ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   745 28759  0  79   4 26000 63437 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   483 28759  0  79   4 26036 63434 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15207 28759  0  79   4 26072 63593 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31696 28759  0  79   4 26116 63546 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14469 28759  0  79   4 26128 63467 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  2153 28759  0  79   4 26136 63454 semtim ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 30883 28759  0  79   4 26144 63765 flock_ ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   728 28759  0  79   4 26156 63462 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   747 28759  0  79   4 26164 63507 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15460 28759  0  79   4 26208 63324 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1501 28759  0  80   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14870 28759  0  79   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15462 28759  0  79   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16674 28759  0  80   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31646 28759  0  82   4 26232 63586 -      ?          0:02 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31683 28759  0  80   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31703 28759  0  80   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32366 28759  0  80   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32414 28759  0  80   4 26232 63586 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14744 28759  0  79   4 26260 63495 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14649 28759  0  79   4 26288 63562 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14716 28759  0  79   4 26292 63512 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32436 28759  0  81   4 26300 63590 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   744 28759  0  80   4 26324 63592 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31734 28759  0  79   4 26344 63538 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14462 28759  0  79   4 26364 63562 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   741 28759  0  79   4 26376 63543 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16525 28759  0  80   4 26392 63597 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   693 28759  0  79   4 26396 63504 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16326 28759  0  79   4 26432 63509 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1471 28759  0  79   4 26448 63514 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16563 28759  0  79   4 26468 63505 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1433 28759  0  79   4 26492 63527 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1456 28759  0  81   4 26532 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32458 28759  0  80   4 26536 63597 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14691 28759  0  79   4 26552 63569 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1462 28759  0  79   4 26556 63548 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13932 28759  0  79   4 26556 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14479 28759  0  79   4 26564 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 17077 28759  0  80   4 26576 63594 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31638 28759  0  79   4 26604 63572 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14749 28759  0  79   4 26616 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16583 28759  0  80   4 26620 63593 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14000 28759  0  79   4 26628 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14467 28759  0  80   4 26636 63596 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31681 28759  0  80   4 26640 63594 -      ?          0:02 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16022 28759  0  79   4 26648 63594 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13973 28759  0  79   4 26652 64093 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 30886 28759  0  80   4 26656 63603 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32434 28759  0  79   4 26656 63586 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1494 28759  1  80   4 26664 63594 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14287 28759  0  79   4 26664 63569 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   691 28759  0  80   4 26712 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31650 28759  0  80   4 26724 63596 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   736 28759  0  80   4 26780 63597 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 30869 28759  0  80   4 26808 63618 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15481 28759  0  80   4 26812 63597 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   484 28759  0  81   4 26832 63630 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15209 28759  0  80   4 26872 63603 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 12893 28759  0  79   4 26888 63602 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   737 28759  0  79   4 26920 63624 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32674 28759  0  80   4 26936 63609 -      ?          0:02 /usr/local/apache/bin/httpd -k start -DSSL
S    99 12881 28759  0  79   4 26944 63606 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32374 28759  0  80   4 27052 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 15211 28759  0  79   4 27072 63596 -      ?          0:02 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14012 28759  0  79   4 27136 63595 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16617 28759  0  80   4 27296 63597 -      ?          0:02 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14686 28759  0  79   4 27404 64119 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13996 28759  0  79   4 31636 64538 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   688 28759  0  79   4 32408 65083 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 12950 28759  0  79   4 33804 65359 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   906 28759  0  80   4 36128 65929 648624 ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31667 28759  0  79   4 40348 66994 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14478 28759  0  79   4 42072 67431 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14655 28759  0  79   4 42416 67681 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16529 28759  0  80   4 42528 67682 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13933 28759  0  79   4 42616 67558 flock_ ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99   711 28759  0  80   4 42792 67626 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31641 28759  0  79   4 43060 67691 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31799 28759  0  80   4 43064 67635 -      ?          0:02 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32196 28759  0  80   4 43748 67798 flock_ ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1439 28759  0  79   4 44864 68158 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16558 28759  0  79   4 45396 68530 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14742 28759  0  79   4 45404 68227 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 32394 28759  0  79   4 45464 68532 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 30881 28759  0  79   4 45492 68537 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 17245 28759  0  79   4 45748 68539 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99  1502 28759  0  79   4 45864 68537 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14718 28759  0  79   4 46140 68679 -      ?          0:03 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14748 28759  0  79   4 46196 68610 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 13931 28759  0  79   4 46272 68619 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14692 28759  0  79   4 46340 68605 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99   431 28759  0  79   4 46452 68609 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 16030 28759  0  79   4 46512 68755 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 14735 28759  0  79   4 46752 68821 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31207 28759  0  79   4 46800 68771 -      ?          0:00 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31377 28759  0  79   4 46852 69204 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL
S    99 31647 28759  0  79   4 47192 68703 -      ?          0:01 /usr/local/apache/bin/httpd -k start -DSSL


top - 08:33:03 up 89 days, 19:19,  2 users,  load average: 0.19, 0.34, 0.20
Tasks: 274 total,   1 running, 271 sleeping,   1 stopped,   1 zombie
Cpu(s):  0.2%us, 22.4%sy,  3.7%ni, 73.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4047632k total,  4009064k used,    38568k free,   119372k buffers
Swap:  4096532k total,    63948k used,  4032584k free,  1440936k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1438 nobody    20   4  248m  26m 5796 S 35.6  0.7   0:01.52 httpd
 2183 nobody    20   4  243m  22m 4680 S 25.9  0.6   0:00.91 httpd
 2154 nobody    20   4  242m  22m 4748 S 21.9  0.6   0:00.75 httpd
 2181 nobody    20   4  245m  22m 5288 S 14.3  0.6   0:00.49 httpd
 2147 nobody    20   4  242m  21m 4200 S  3.0  0.6   0:00.94 httpd
 2180 nobody    19   4  245m  23m 4936 S  1.0  0.6   0:00.21 httpd
 1499 nobody    20   4  247m  25m 5196 S  0.7  0.6   0:00.41 httpd
 2195 root      15   0 12876 1240  808 R  0.3  0.0   0:00.09 top
13927 root      34  19 17584 1168  824 S  0.3  0.0   0:16.41 mysqldump
    1 root      15   0 10352  560  524 S  0.0  0.0   0:15.41 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:15.89 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:01.22 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:07.47 migration/1
    6 root      34  19     0    0    0 S  0.0  0.0   0:01.71 ksoftirqd/1
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:11.11 migration/2
    9 root      34  19     0    0    0 S  0.0  0.0   1:29.29 ksoftirqd/2
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:05.38 migration/3
   12 root      34  19     0    0    0 S  0.0  0.0   0:10.27 ksoftirqd/3
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3
   14 root      10  -5     0    0    0 S  0.0  0.0   0:01.52 events/0
   15 root      10  -5     0    0    0 S  0.0  0.0   0:01.57 events/1
   16 root      10  -5     0    0    0 S  0.0  0.0   0:01.30 events/2
   17 root      10  -5     0    0    0 S  0.0  0.0   0:00.65 events/3
   18 root      10  -5     0    0    0 S  0.0  0.0   0:01.68 khelper
   57 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
   64 root      10  -5     0    0    0 S  0.0  0.0   0:02.47 kblockd/0
   65 root      10  -5     0    0    0 S  0.0  0.0   0:00.69 kblockd/1
   66 root      10  -5     0    0    0 S  0.0  0.0   0:24.51 kblockd/2
   67 root      10  -5     0    0    0 S  0.0  0.0   0:07.52 kblockd/3
   68 root      15  -5     0    0    0 S  0.0  0.0   0:00.0
   

   top - 08:33:53 up 89 days, 19:20,  2 users,  load average: 0.08, 0.28, 0.19
Tasks: 281 total,   1 running, 277 sleeping,   1 stopped,   2 zombie
Cpu(s):  0.0%us,  1.2%sy,  3.0%ni, 95.5%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   4047632k total,  3993636k used,    53996k free,   117724k buffers
Swap:  4096532k total,    63948k used,  4032584k free,  1407884k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2223 nobody    20   4  243m  22m 4344 S  4.3  0.6   0:00.14 httpd
 2214 nobody    19   4  243m  22m 4304 S  4.0  0.6   0:00.14 httpd
 2154 nobody    19   4  243m  22m 4824 S  3.3  0.6   0:00.87 httpd
 2147 nobody    19   4  242m  21m 4200 S  3.0  0.6   0:01.51 httpd
 2183 nobody    20   4  244m  22m 5184 S  1.3  0.6   0:00.99 httpd
 2213 root      15   0 12876 1236  808 R  0.3  0.0   0:00.07 top
13927 root      34  19 17584 1168  824 S  0.3  0.0   0:16.74 mysqldump
31191 root      15   0 94244 3412 2652 S  0.3  0.1   0:00.04 sshd
    1 root      15   0 10352  560  524 S  0.0  0.0   0:15.41 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:15.89 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:01.22 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:07.47 migration/1
    6 root      34  19     0    0    0 S  0.0  0.0   0:01.71 ksoftirqd/1
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:11.11 migration/2
    9 root      34  19     0    0    0 S  0.0  0.0   1:29.29 ksoftirqd/2
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:05.38 migration/3
   12 root      34  19     0    0    0 S  0.0  0.0   0:10.27 ksoftirqd/3
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3
   14 root      10  -5     0    0    0 S  0.0  0.0   0:01.52 events/0
   15 root      10  -5     0    0    0 S  0.0  0.0   0:01.57 events/1
   16 root      10  -5     0    0    0 S  0.0  0.0   0:01.30 events/2
   17 root      10  -5     0    0    0 S  0.0  0.0   0:00.65 events/3
   18 root      10  -5     0    0    0 S  0.0  0.0   0:01.68 khelper
   57 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
   64 root      10  -5     0    0    0 S  0.0  0.0   0:02.47 kblockd/0
   65 root      10  -5     0    0    0 S  0.0  0.0   0:00.69 kblockd/1
   66 root      10  -5     0    0    0 S  0.0  0.0

Open in new window

Avatar of MrNed
MrNed
Flag of Australia image

Based on what you provided, do nothing. That only shows CPU which is fine. You need to look at your memory and disk I/O stats too. I assume since you're asking the question you have noticed performance problems? If so what were the symptoms?
Well, your health check looks really good.  I really don't think you need to upgrade at this time.  Are you experiencing some signs of performance hit?  Don't be fooled about the free memory.  Linux does an awesome job of memory management and it uses all the memory.  You got a really nice setup there and I'd load it down even more.  Unless things are breaking down, you are set for quite sometime.. unless there is some kind of load that I don't know about.  You gotta a kickin' processor and plenty of RAM.  You can increase your RAM but you are not really going to see a gain in performance, I just don't see that kind of load.  Did you build that system?  Some nice hardware, you should be running smooth for sometime.
Avatar of David
iostat gives you I/O performance.  You certainly don't have a CPU or memory bottleneck based on just this info.

If you are not already doing so, then reconfigure the disk to use the md driver so you have software-based RAID1.  (THen buy additional disks as necessary).  LINUX will read-balance I/O in RAID1, so in perfect world, you get twice the IOPs on reads on a mirrored pair then a standalone disk.   Of course if I/O is not a bottleneck, then only upgrade I can think of is making darned sure you have a good backup hardware, and perhaps another network card for either redundancy or bonding to double throughput.
What is the symtom you are trying to overcome?
Web servers are rarely the bottleneck, especially without .NET
Avatar of melwong
melwong

ASKER

The symptom is our sites are pretty darn slow during that time. I suspect it could be mysql but i just converted some tables to InnoDB and indexed some, but still is slow. It only improved a bit.

For instance, I see AT LEAST 20 of these lines when u run "mysqladmin processlist"

Query          | 1683 | Locked                   | update matches_details set status='Full-time' where match_id='217671'  

But i already converted matches_details into InnoDB and indexed the table.

looking at the last two top output two things i have noticed.

1. why mysqldump is there running in both the outputs.
2. httpd is consuming so much of cpu. you should check the pages which might be causing this and fine tune the application. maybe you can try hitting the pages one by one and check the load it generates on the system resources. this will give you some idea on what and where is wrong.
ASKER CERTIFIED SOLUTION
Avatar of ivailoj
ivailoj
Flag of Bulgaria image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Normally if your web pages are loading slowly it's because there is somethign on the page slowing them down.
usually it's a database issue.

On pages where data doesn't change alot. You should look into web caching. and only updating when there's a change.

you have a pretty decent server there. it shouldn't be slow at all.
If you want some more speed from your server look into solid state drives.
Avatar of melwong

ASKER

Yes, my site is gzip. It's just that during 3am till 10am (GMT+8) the site will be slow. Other than that, it's pretty darn fast. How can i systematically identify which page, sql creates the load?
I would try using a page execution script.
store the values in a table somewhere. keep track of the page load time and time of day.
then you can run daily stats to see which pages are the slowest.
You still have yet to return any diagnostic information that reveals your I/O overhead.  Not much point to trying to optimize database if you don't know how much I/O activity there is.  

You can choose from simple iostat, sar, sag ..

A really nice one that is more or less a superset of top, but also does equivalent for disk and file system Io is nmon

Just do apt-get install nmon

then run it and be amazed at how much it will tell you about pretty much all aspects of overhead

In interest of full disclosure, and just to say something nice about what used to be the evil company, IBM, they wrote nmon and graciously open sourced it, The home page is below, where you can get latest build.
http://www.ibm.com/developerworks/aix/library/au-analyze%5Faix/
Avatar of melwong

ASKER

Here're my disk io for apache and mysql servers
Disk IO of my web server (apache)

root@server1 [~]# iostat
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.24    2.36    3.26    3.50    0.00   90.64

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              51.63      4036.55      2223.21  278651626  153473288
sda1              0.00         0.03         0.00       2234         16
sda2             12.17      1939.44      2053.84  133883824  141781312
sda3              3.08        33.42        43.66    2307397    3013720
sda4              0.00         0.00         0.00         16          0
sda5              7.39        28.47        74.50    1965130    5143192
sda6              0.00         0.03         0.00       1755        264
sda7              0.18         3.91         0.20     270154      14112
sda8              1.48         0.18        25.20      12386    1739728
sda9             27.34      2031.05        25.80  140208130    1780944

root@server1 [~]# sar
12:00:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM       all      0.18      3.18      4.15      0.11      0.00     92.38
12:20:01 AM       all      0.20      2.26      2.41      0.05      0.00     95.08
12:30:01 AM       all      0.61      1.79      3.63      0.15      0.00     93.81
12:40:01 AM       all      0.17      2.49      4.36      0.12      0.00     92.87
12:50:01 AM       all      0.28      1.89      2.80      0.15      0.00     94.88
01:00:01 AM       all      0.24      1.82      2.08      0.12      0.00     95.75
01:10:02 AM       all      0.17      4.19      4.66     21.93      0.00     69.05
01:20:02 AM       all      0.17      2.19      2.87     24.20      0.00     70.58
01:30:01 AM       all      0.17      2.07      3.83     24.49      0.00     69.44
01:40:02 AM       all      0.16      2.25      5.25     27.05      0.00     65.29
01:50:04 AM       all      0.17      1.66      5.30     47.10      0.00     45.77
02:00:05 AM       all      0.18      2.23      5.15     46.47      0.00     45.97
02:10:01 AM       all      0.25      2.24      5.48     27.07      0.00     64.96
02:20:01 AM       all      0.17      2.11      3.57     24.82      0.00     69.34
02:30:02 AM       all      0.17      1.85      3.18     24.35      0.00     70.46
02:40:07 AM       all      0.16      1.81      5.01     27.45      0.00     65.57
02:50:02 AM       all      0.17      2.35      5.25     41.39      0.00     50.83
03:00:02 AM       all      0.18      1.85      4.36     25.06      0.00     68.54
03:10:01 AM       all      0.16      2.28      5.30      1.31      0.00     90.95
03:20:01 AM       all      0.16      1.51      2.16      0.31      0.00     95.85
03:30:01 AM       all      0.16      2.04      3.46      0.22      0.00     94.12
03:40:01 AM       all      0.92      1.77      4.25      0.25      0.00     92.80
03:50:01 AM       all      0.24      1.81      2.48      3.16      0.00     92.32
04:00:01 AM       all      0.19      2.35      2.62      1.18      0.00     93.66
04:10:01 AM       all      0.68      2.43      4.99      7.18      0.00     84.72
04:20:01 AM       all      0.17      1.88      3.31      0.29      0.00     94.35
04:30:01 AM       all      0.16      1.76      1.56      0.09      0.00     96.42
04:40:01 AM       all      0.17      1.37      1.41      0.13      0.00     96.92
04:50:01 AM       all      0.36      1.41      2.03      0.58      0.00     95.62
05:00:01 AM       all      0.16      1.81      2.89      0.14      0.00     94.99
05:10:01 AM       all      0.17      2.22      5.19      0.12      0.00     92.29
05:20:01 AM       all      0.16      1.78      2.85      0.14      0.00     95.07
05:30:01 AM       all      0.16      1.71      1.94      0.24      0.00     95.95
05:40:01 AM       all      0.16      2.30      4.86      0.24      0.00     92.45
05:50:01 AM       all      0.34      1.72      2.77      0.46      0.00     94.72
06:00:01 AM       all      0.17      1.40      2.25      0.15      0.00     96.03
06:10:01 AM       all      0.17      2.18      4.48      0.16      0.00     93.01
06:20:01 AM       all      0.16      1.81      2.26      0.15      0.00     95.62
06:30:01 AM       all      0.16      2.06      2.98      0.12      0.00     94.67
06:40:01 AM       all      0.17      1.98      4.16      0.12      0.00     93.57
06:50:01 AM       all      0.34      1.70      2.79      0.34      0.00     94.83
07:00:01 AM       all      0.17      1.65      2.22      0.14      0.00     95.83

07:00:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
07:10:01 AM       all      0.17      1.85      4.33      0.14      0.00     93.51
07:20:01 AM       all      0.17      1.91      2.55      0.12      0.00     95.26
07:30:01 AM       all      0.16      1.47      1.77      0.08      0.00     96.51
07:40:01 AM       all      0.16      1.88      4.01      0.17      0.00     93.78
07:50:01 AM       all      0.34      1.88      3.09      0.49      0.00     94.20
08:00:01 AM       all      0.17      1.69      1.94      0.10      0.00     96.10
08:10:01 AM       all      0.17      2.38      5.36      0.19      0.00     91.90
08:20:01 AM       all      0.17      1.81      2.33      0.18      0.00     95.51
08:30:01 AM       all      0.17      2.62      2.39      0.21      0.00     94.61
08:40:01 AM       all      0.17      2.43      4.03      0.15      0.00     93.22
08:50:01 AM       all      0.35      2.30      2.94      2.22      0.00     92.19
Average:          all      0.22      2.03      3.46      7.23      0.00     87.06

_______________________________________________________________________________________

Disk IO for MySQL Server

root@server2 [~]# iostat
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.43    1.82    1.00    1.90    0.00   93.85

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              13.00       424.22       168.01   19752331    7822695
sda1              0.00         0.05         0.00       2538         80
sda2              0.35         0.18        67.71       8613    3152937
sda3             10.57       394.81        68.70   18383269    3198690
sda4              0.00         0.00         0.00          8          0
sda5              1.30        24.26        21.40    1129754     996448
sda6              0.00         0.03         0.01       1243        272
sda7              0.53         2.90         8.66     135202     403352
sda8              0.00         0.02         0.00        816          0
sda9              0.25         1.94         1.52      90280      70916

root@server2 [~]# sar
12:00:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM       all      1.66      1.12      1.06      1.99      0.00     94.17
12:20:01 AM       all      1.54      0.79      0.76      1.63      0.00     95.29
12:30:01 AM       all      1.18      0.59      1.59      2.08      0.00     94.56
12:40:01 AM       all      1.73      0.69      0.72      1.69      0.00     95.18
12:50:01 AM       all      1.70      0.67      0.57      1.57      0.00     95.49
01:00:01 AM       all      1.52      4.54      4.11      1.04      0.00     88.79
01:10:01 AM       all      2.21     21.92      3.48     10.69      0.00     61.70
01:20:02 AM       all      1.46     13.73      1.02      9.35      0.00     74.44
01:30:01 AM       all      1.36      0.82      0.47      1.58      0.00     95.76
01:40:01 AM       all      1.34      1.18      0.43      1.26      0.00     95.79
01:50:01 AM       all      1.66      1.00      0.53      2.13      0.00     94.68
02:00:01 AM       all      1.46      1.88      0.95      1.70      0.00     94.01
02:10:01 AM       all      0.80      1.19      0.68      1.86      0.00     95.47
02:20:01 AM       all      1.33      1.56      0.87      1.42      0.00     94.82
02:30:01 AM       all      1.05      1.42      0.70      1.50      0.00     95.33
02:40:01 AM       all      0.93      1.38      0.64      1.41      0.00     95.65
02:50:01 AM       all      0.82      1.16      0.75      1.24      0.00     96.03
03:00:01 AM       all      1.22      0.96      0.48      1.51      0.00     95.83
03:10:01 AM       all      0.38      2.16      0.62      1.53      0.00     95.31
03:20:01 AM       all      0.29      1.81      0.44      1.44      0.00     96.02
03:30:01 AM       all      0.93      2.33      0.70      2.39      0.00     93.66
03:40:01 AM       all      0.49      1.95      0.57      1.49      0.00     95.49
03:50:01 AM       all      0.94      1.98      0.75      1.26      0.00     95.06
04:00:01 AM       all      0.72      2.29      0.83      1.28      0.00     94.88
04:10:01 AM       all      1.32      2.12      0.92      2.91      0.00     92.73
04:20:01 AM       all      0.47      2.22      0.62      1.40      0.00     95.29
04:30:01 AM       all      0.74      1.86      0.76      1.38      0.00     95.26
04:40:01 AM       all      0.79      1.69      0.61      1.25      0.00     95.66
04:50:01 AM       all      0.47      1.18      0.51      1.08      0.00     96.76
05:00:01 AM       all      0.54      1.59      0.61      1.29      0.00     95.97
05:10:01 AM       all      0.73      2.50      0.76      1.33      0.00     94.68
05:20:01 AM       all      0.63      1.68      0.52      1.23      0.00     95.94
05:30:01 AM       all      0.59      1.83      0.71      1.31      0.00     95.56
05:40:01 AM       all      0.46      2.15      0.65      1.24      0.00     95.49
05:50:01 AM       all      0.49      1.75      0.64      1.34      0.00     95.78
06:00:01 AM       all      0.40      1.76      0.77      1.45      0.00     95.62
06:10:01 AM       all      1.08      2.62      0.67      1.60      0.00     94.03
06:20:01 AM       all      1.03      3.77      0.93      1.33      0.00     92.93
06:30:01 AM       all      1.01      3.90      0.74      1.33      0.00     93.02
06:40:01 AM       all      1.16      2.98      0.65      1.44      0.00     93.77
06:50:01 AM       all      0.99      2.68      0.87      1.23      0.00     94.23
07:00:01 AM       all      0.56      1.82      0.71      1.39      0.00     95.51

07:00:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
07:10:01 AM       all      0.58      1.74      0.57      1.39      0.00     95.72
07:20:01 AM       all      0.48      1.87      0.77      1.26      0.00     95.62
07:30:01 AM       all      0.21      0.84      0.37      1.21      0.00     97.37
07:40:01 AM       all      0.69      2.45      0.81      1.33      0.00     94.71
07:50:01 AM       all      0.78      1.97      0.88      1.44      0.00     94.93
08:00:01 AM       all      0.82      1.55      0.60      1.20      0.00     95.83
08:10:01 AM       all      0.69      1.83      0.67      1.45      0.00     95.37
08:20:01 AM       all      0.28      2.24      0.66      1.16      0.00     95.66
08:30:01 AM       all      0.15      2.95      0.98      1.19      0.00     94.73
08:40:01 AM       all      0.11      2.07      0.64      1.33      0.00     95.85
08:50:01 AM       all      0.19      2.18      0.78      1.39      0.00     95.46
Average:          all      0.89      2.47      0.83      1.79      0.00     94.02
root@server2 [~]#

Open in new window

To see what happened on SQL:
username is your username
password is yopur password

mysqladmin proc stat -uusername -ppassword

Look for:
Slow queries: ???
if number is more than 50 for let say 1 hours it is bad.

Look at: http://dev.mysql.com/doc/refman/5.5/en/slow-query-log.html

and enable this log!
After that it is interesting what will MySQL say :)?

Please paste some of Heavy queries just for fun.
Geez - why is sda partitioned into 9 slices?  This is hurting you.

You want better performance, simple, buy more HDDs, start by moving the sda9 slice to the new HD ( I suggest using md driver, and going RAID1, so then you would then define md0 as 2 disks in RAID1, md1 as 2 disks in raid1.

sda9 is getting approx half of the read traffic, so I would start by moving that to the new disks.
P.S. you have plenty of memory & cpu.  The dumps confirm your problem is I/O.  The cure is more disk drives.  

(But here is a sneaky thing, if you have a few hot files that are constantly being read, and not written after the system boots up, then move them to the /dev/shm directory on bootup, and point the application there).  the /dev/shm directory is a ramdisk.  

You will see a profound performance increase if you modify some of your sql code to create scratch index tables and such in /dev/shm instead of one of the file systems.   One of the things I do in my professional life is write the enclosure/hardware management code for some well-known NAS appliance vendors. When I started moving just my code to /dev/shm as part of the bootup sequence, it effectively removed all measurable overhead.  You very well might see overall performance double or triple by just doing that alone.
Avatar of melwong

ASKER

dlethe, if i would put my website's images, css, js in /dev/shm, how do my site refers to them (i mean the url).
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of melwong

ASKER

thanks guyd