Pardon my brain dump. The problem has a lot of information and I'm just trying to get it all out there, first:
I've built a SAN using OF 2.99.2. I've updated it and configured the FC and ISCSI to connect to a volume. When I perform ISCSI via ethernet backups, I get horrendous throughput. (20MB/s from one host, 5-10MB/s from another..) I noticed that even though I set 9000 MTU on the NIC, it's actually set at 7000 and denies any change. I'm assuming the NIC (some booty rltk junker) doesn't support over 7000MTu..
I can copy full speed over the Fibre channel to the same volume (~2.5-3.0Gbps).
I can copy full speed to a different ISCSI (starwind from my workstation) over ethernet host (95MB/s - Gigabit, 9000MTU)
When I copy using ISCSI over ethernet to the OF host to the same volume I get max speed, it crawls at 5-10-20MB/s.
The volume is set up to use FILE i/o
I can mount one of the VMDKs the backup script just previously wrote on the same 'slow' volume via ISCSI (while it's backing up the rest of the vms at the slow speed), run bench32 on it and still achieve 40-50MB/s. Perplexing.
It has to be the way ISCSI is configured on the OF box. Has to be.... Is there anything I'm just not thinking of?