amigan_99
asked on
Netapp FAS 3140 - help interpreting lots of cache hits
We have been having some serious disk performance problems. During certain operations in VMWare like storage vmotion or other the ESX hosts and their guest OS's lose access to the hard drive which are reached via iSCSI. Cisco UCS VMware 5 communicating with the 3140 over 10Gbps links. No errors on the ethernet path. But seeing cache hits at or near 100% all the time. What does this indicate?
heraklesb> sysstat -u 1
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
10% 915 8719 864 724 0 0 0 26s 100% 0% - 9%
7% 955 7625 1181 544 8 0 0 26s 98% 0% - 9%
12% 819 8833 1074 19501 631 0 0 26s 100% 6% T 15%
49% 1130 9214 1858 82880 79888 0 0 26s 100% 100% : 100%
17% 812 7228 423 32576 11368 0 0 26s 99% 100% : 33%
17% 1083 9123 774 16732 40248 0 0 26s 99% 100% : 53%
38% 3557 9541 297324 3772 23548 0 0 26s 100% 72% : 35%
22% 2656 5399 237799 936 0 0 0 26s 100% 0% - 11%
7% 851 6582 3580 940 24 0 0 26s 94% 0% - 7%
6% 512 4146 557 920 0 0 0 26s 97% 0% - 10%
5% 384 2641 309 440 0 0 0 26s 97% 0% - 6%
9% 858 7569 449 396 32 0 0 26s 98% 0% - 7%
8% 876 6573 1601 576 0 0 0 26s 98% 0% - 8%
7% 669 6293 426 1096 0 0 0 26s 95% 0% - 11%
5% 408 4215 602 492 24 0 0 26s 89% 0% - 7%
4% 388 3145 442 268 0 0 0 26s 100% 0% - 7%
23% 1472 13173 1425 17052 12556 0 0 26s 100% 26% T 25%
38% 973 9131 386 80056 46725 0 0 26s 100% 100% : 98%
14% 618 5120 257 17252 18124 0 0 26s 100% 100% : 35%
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
14% 568 4759 725 17104 35692 0 0 26s 98% 100% : 46%
heraklesb> sysstat -u 1
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
39% 325 2805 740 436 0 0 0 13s 86% 0% - 6%
10% 1166 9126 3180 1256 24 0 0 13s 100% 0% - 10%
8% 1178 6791 4929 1676 0 0 0 13s 99% 0% - 12%
4% 291 3147 360 232 0 0 0 13s 100% 0% - 4%
16% 334 2213 1011 18440 15380 0 0 13s 99% 43% T 42%
19% 497 8507 385 55052 32376 0 0 13s 100% 100% : 83%
18% 986 9137 1125 25400 27236 0 0 13s 100% 100% : 43%
12% 507 4202 467 3700 35208 0 0 13s 100% 100% : 46%
9% 1068 8792 1809 748 4136 0 0 13s 99% 22% : 21%
9% 980 6072 41322 208 8 0 0 13s 100% 0% - 5%
10% 998 2014 77318 412 24 0 0 13s 100% 0% - 11%
5% 561 5180 478 256 0 0 0 13s 99% 0% - 5%
16% 1633 3090 149573 188 0 0 0 13s 100% 0% - 5%
9% 946 8522 20868 672 24 0 0 13s 100% 0% - 10%
7% 801 8548 518 504 8 0 0 13s 100% 0% - 8%
6% 499 4292 300 148 0 0 0 13s 100% 0% - 5%
7% 678 6066 486 444 24 0 0 13s 98% 0% - 6%
6% 604 6306 745 420 0 0 0 38s 100% 0% - 7%
32% 448 2958 947 67600 46508 0 0 39s 100% 77% T 75%
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
23% 332 2346 161 58888 18884 0 0 39s 100% 100% : 51%
15% 770 6636 875 404 17880 0 0 39s 100% 100% : 21%
16% 1009 10627 530 3272 44408 0 0 39s 100% 97% : 51%
4% 338 2424 126 692 24 0 0 39s 99% 0% - 10%
4% 465 3745 292 140 0 0 0 39s 100% 0% - 3%
7% 685 7148 1934 856 8 0 0 39s 99% 0% - 12%
7% 749 8642 2026 1708 24 0 0 39s 98% 0% - 15%
9% 856 2762 58752 160 0 0 0 11s 100% 0% - 6%
6% 575 2047 38734 456 0 0 0 11s 99% 0% - 5%
6% 458 1996 18820 632 24 0 0 11s 96% 0% - 25%
heraklesb> sysstat -u 1
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
10% 915 8719 864 724 0 0 0 26s 100% 0% - 9%
7% 955 7625 1181 544 8 0 0 26s 98% 0% - 9%
12% 819 8833 1074 19501 631 0 0 26s 100% 6% T 15%
49% 1130 9214 1858 82880 79888 0 0 26s 100% 100% : 100%
17% 812 7228 423 32576 11368 0 0 26s 99% 100% : 33%
17% 1083 9123 774 16732 40248 0 0 26s 99% 100% : 53%
38% 3557 9541 297324 3772 23548 0 0 26s 100% 72% : 35%
22% 2656 5399 237799 936 0 0 0 26s 100% 0% - 11%
7% 851 6582 3580 940 24 0 0 26s 94% 0% - 7%
6% 512 4146 557 920 0 0 0 26s 97% 0% - 10%
5% 384 2641 309 440 0 0 0 26s 97% 0% - 6%
9% 858 7569 449 396 32 0 0 26s 98% 0% - 7%
8% 876 6573 1601 576 0 0 0 26s 98% 0% - 8%
7% 669 6293 426 1096 0 0 0 26s 95% 0% - 11%
5% 408 4215 602 492 24 0 0 26s 89% 0% - 7%
4% 388 3145 442 268 0 0 0 26s 100% 0% - 7%
23% 1472 13173 1425 17052 12556 0 0 26s 100% 26% T 25%
38% 973 9131 386 80056 46725 0 0 26s 100% 100% : 98%
14% 618 5120 257 17252 18124 0 0 26s 100% 100% : 35%
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
14% 568 4759 725 17104 35692 0 0 26s 98% 100% : 46%
heraklesb> sysstat -u 1
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
39% 325 2805 740 436 0 0 0 13s 86% 0% - 6%
10% 1166 9126 3180 1256 24 0 0 13s 100% 0% - 10%
8% 1178 6791 4929 1676 0 0 0 13s 99% 0% - 12%
4% 291 3147 360 232 0 0 0 13s 100% 0% - 4%
16% 334 2213 1011 18440 15380 0 0 13s 99% 43% T 42%
19% 497 8507 385 55052 32376 0 0 13s 100% 100% : 83%
18% 986 9137 1125 25400 27236 0 0 13s 100% 100% : 43%
12% 507 4202 467 3700 35208 0 0 13s 100% 100% : 46%
9% 1068 8792 1809 748 4136 0 0 13s 99% 22% : 21%
9% 980 6072 41322 208 8 0 0 13s 100% 0% - 5%
10% 998 2014 77318 412 24 0 0 13s 100% 0% - 11%
5% 561 5180 478 256 0 0 0 13s 99% 0% - 5%
16% 1633 3090 149573 188 0 0 0 13s 100% 0% - 5%
9% 946 8522 20868 672 24 0 0 13s 100% 0% - 10%
7% 801 8548 518 504 8 0 0 13s 100% 0% - 8%
6% 499 4292 300 148 0 0 0 13s 100% 0% - 5%
7% 678 6066 486 444 24 0 0 13s 98% 0% - 6%
6% 604 6306 745 420 0 0 0 38s 100% 0% - 7%
32% 448 2958 947 67600 46508 0 0 39s 100% 77% T 75%
CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk
ops/s in out read write read write age hit time ty util
23% 332 2346 161 58888 18884 0 0 39s 100% 100% : 51%
15% 770 6636 875 404 17880 0 0 39s 100% 100% : 21%
16% 1009 10627 530 3272 44408 0 0 39s 100% 97% : 51%
4% 338 2424 126 692 24 0 0 39s 99% 0% - 10%
4% 465 3745 292 140 0 0 0 39s 100% 0% - 3%
7% 685 7148 1934 856 8 0 0 39s 99% 0% - 12%
7% 749 8642 2026 1708 24 0 0 39s 98% 0% - 15%
9% 856 2762 58752 160 0 0 0 11s 100% 0% - 6%
6% 575 2047 38734 456 0 0 0 11s 99% 0% - 5%
6% 458 1996 18820 632 24 0 0 11s 96% 0% - 25%
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER