Why is my LSI raid so slow?

make, model of drives.

https://www.broadcom.com/products/storage/raid-controllers/megaraid-sas-9361-8i

ST8000VN0022

seagate

Go back a bit

So, can you please tell me the make, model and generation of server, can you please provide the processor that's in there, can you please also tell me if you have any other RAID controllers in there, what Operating system is it and is this a new install or an existing build?

Thanks
Alex

Is there a defective drive in the array?

Do you have the CacheVault® Flash cache protection module (battery or supercap)? Without that the write cache is disabled by default which will give very poor performance in RAID 5 and 6. How big is your cache module, they do 1 and 2 GB versions.

RAID 5/6 has slow writes but fast reads.  RAID 1 is faster, a good balance is raid 10.  I'm guessing you are doing 80% write and 20% read.

IO penalty (read) = 1/1 (one RAID IO per each host IO)
IO penalty (write) = 4/1 (4 RAID IOs per each host IO)

Windows 10 - Gen9   Coffee Lake i7 3.2 GHz 6 core Asus MB ROG STRIX B360-g Gaming 32 GB RAM

The impact of bad drives is:
All rations are RadIO's vs. Host IO's as David used (assuming you have a 8 way raid system).
Read:
RAID 5/6:   block is on OK Disk              1 / 1
                    block is on bad disk            7/1 read all disks, and calculate the missing block = 7/1
RAID 1        always                                   1/1
                   
Write:
RAID 5/6:   block is on OK Disk  & Chksum is on OK disk           4 / 1
                    block is on bad disk & Checksum is on OK disk      8 / 1
                    block is on OK disk & Checksum is on bad Disk      1 / 1
RAID 1        always                                   (1/1 RAID IO/HostIO)

(Raid 1, valid RaidArray) = 2/1 (RAID IO / HostIO).  Can be done in parallel. no calculations involved.

8TB (56TB) data set...

While the drive reflects raid optimization for use in Raid Nas devices, whether it would work as flawlessly with the controller.

Other comments related to cache ..

Checking the log of the controller may shed light on performance.

Note your single drive test, 100Mb on a 6gb link suggests the performance of the drive is contributing to a lower throughput.

You can always check if it is the lack of BBU that's slowing it down with the following command:

MegaCli64.exe -LDSetProp CachedBadBBU -L0 -a0

If that speeds it up run "MegaCli64 -LDSetProp NoCachedBadBBU -L0 -a0" and order the cache backup battery.

If you can't find megacli.exe download on Broadcom's site it is on Lenovo's.

Can you say "you bought the wrong raid controller!"?

Here is the answer from Broadcom:

The 9341 does not have a RAID processor onboard nor does it have cache memory.
It is an entry level controller that is not as fast as the 9361 controller.

RAID 0, 1 and 10 will give you better performance.
Make sure that you use MegaRAID Storage Manager to enable the drive cache.

Go to MegaRAID Storage Manager logical tab and right click the Virtual drive and choose Set Virtual Drive Properties.
Enable Disk Cache.

Thank you and let me know if you have any questions, Storage Technical support Broadcom Inc.

They told you to enable the drive cache without mentioning that it risks data loss? That's disgraceful.

I had to guess at the controller you had, thought it was a 9361 since there's no 9381 which is what you originally put.

RAID 5 especially in your case with use of 8TB is not a wise choice.

Long ago article describes the issue faced with a single failure in a raid 5 config:
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

After replacing the SAS9341-8i with a SAS9361-8i I saw a huge performance boost. Now the performance runs around 250 to 400 MB/sec.
Thank you, everyone, for your suggestions and help.

IMHO in this case selecting your own comment as the solution without awarding point to the other comments that led to this decision is rather unkind.