MEATBALLHERO
asked on
iSCSI Speed - Why so inconsistant?
I have created tow SANs using OPEN-E for an Active -Active High Availablility Failover. I also Have created a Microsoft 2008 Cluster R2. I have two iscsi ports in each 2008 Server Node.
On Node A - I was able to grab a 7GB file and copy and paste it to my desktop in 12 seconds. It showed 500+ MBs a second.
However on Node B - I am only able to get 120 MBs a second and it takes 45 seconds to copy the same file. The two nodes are the exact same spec, processor, memory, etc...
Nic settings are the same. I used dells recomendation to set up iSCSI network accept for Jumbo frames and the Netsh commands.
I double checked the cards i didn't miss anything. The switches that are used in the high availability setup are the same.
I stopped and restarted the cluster and now Node one no longer copies at 500mbs a second. Now it only copies at 120mbs and takes 45 seconds to copy the same file.
What's weird is when it was copying at 500mbs a second I couln't see any iSCSI traffic on the TASK MANAGER NEtwork monitor tab and it said 0% under iSCSI Nics during the lightning fast copy . Now I can see the graph during the slower copy and it shows about 55%.
Can this be explained. I am using MPIO.
On Node A - I was able to grab a 7GB file and copy and paste it to my desktop in 12 seconds. It showed 500+ MBs a second.
However on Node B - I am only able to get 120 MBs a second and it takes 45 seconds to copy the same file. The two nodes are the exact same spec, processor, memory, etc...
Nic settings are the same. I used dells recomendation to set up iSCSI network accept for Jumbo frames and the Netsh commands.
I double checked the cards i didn't miss anything. The switches that are used in the high availability setup are the same.
I stopped and restarted the cluster and now Node one no longer copies at 500mbs a second. Now it only copies at 120mbs and takes 45 seconds to copy the same file.
What's weird is when it was copying at 500mbs a second I couln't see any iSCSI traffic on the TASK MANAGER NEtwork monitor tab and it said 0% under iSCSI Nics during the lightning fast copy . Now I can see the graph during the slower copy and it shows about 55%.
Can this be explained. I am using MPIO.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Aaron Tomosky
If it's gigabit you can't go over 120MB/s no matter how fast your array may be.
To summarize ... you are actually getting reasonable results, and test A is an invalid test because the O/S is lying about the file really being copied. Nothing is wrong, you are getting what you should be getting.
ASKER
Wait...are you saying that even though it looks like I copied a 7GB file in 12 seconds, thats not true?
Are you saying that even though the window that shows the file copying closes, that the file is still copying in the back ground or something?
Are you saying that even though the window that shows the file copying closes, that the file is still copying in the back ground or something?
Yes, that is exactly what I am saying. Do the math. Let's ignore inefficiency and overhead.
7GB * 8 bits/byte / 12 seconds = 4.7 gigabits/sec
do you even HAVE a 4.7 gigabit connection, or do you have perhaps a 1Gigabit connection?
For that matter, I doubt you even have a disk drive that can read or write 7Gbytes in 12 seconds, let alone read the 7GB in the first 6 seconds, then write it at the other end in 6 seconds ... and, even then, we ignore the physics of the speed of light and the time it takes to move those electrons from point A to point B.
Bottom line, your benchmarks are giving you bad data. Based on the sustained numbers with the longer test, I see no reason for any concern. Sure you can always do some tweaking, but don't expect anything more than +/- 20%
7GB * 8 bits/byte / 12 seconds = 4.7 gigabits/sec
do you even HAVE a 4.7 gigabit connection, or do you have perhaps a 1Gigabit connection?
For that matter, I doubt you even have a disk drive that can read or write 7Gbytes in 12 seconds, let alone read the 7GB in the first 6 seconds, then write it at the other end in 6 seconds ... and, even then, we ignore the physics of the speed of light and the time it takes to move those electrons from point A to point B.
Bottom line, your benchmarks are giving you bad data. Based on the sustained numbers with the longer test, I see no reason for any concern. Sure you can always do some tweaking, but don't expect anything more than +/- 20%
Can you verify the write speed of your desktop's hard drive? And if the drive has built in cache? You may get odd benchmark results if the drive has high speed memory to cache writes to disk, but then has to save that data to disk at regular disk write speeds.
And what nics are in both systems
The NICs don't matter. Benchmarking HAS to be wrong. You are reading 7GB, transferring 7GB, and then writing 7 GB in all of 12 seconds.
That is roughly 2 GigaBYTES per second. It doesn't matter what NIC you have. You'd have to have infiniband, and some rather expensive RAID hardware to obtain this. If you had such hardware you would know it.
That is roughly 2 GigaBYTES per second. It doesn't matter what NIC you have. You'd have to have infiniband, and some rather expensive RAID hardware to obtain this. If you had such hardware you would know it.
ASKER
I have two San Servers that replicate. Both have a 16 port promise raid with 512MB of Cache. There are 16 SAS drives in each SAN that run at 15K RPM. this is where the 7GB file sits. I copied it to the desktop of my host which is 4 SAS raid 10 drives at 15K RPM. The drives all have cache enabled and so does the raid cards on all servers.
I do have a 10GB ethernet but that is connected SAN to SAN for replication and not SAN to Host.
Each San has 4 ISCSI Ports and My Host Has two but I put in the IP addresses of all 4 SAN Ports usining MPIO.
I do have a 10GB ethernet but that is connected SAN to SAN for replication and not SAN to Host.
Each San has 4 ISCSI Ports and My Host Has two but I put in the IP addresses of all 4 SAN Ports usining MPIO.
It doesn't matter. No way can your hardware do 2Gbytes/sec.