Solved

iSCSI Speed - Why so inconsistant?

Posted on 2013-11-27
10
292 Views
Last Modified: 2016-11-23
I have created tow SANs using OPEN-E for an Active -Active High Availablility Failover. I also Have created a Microsoft 2008 Cluster R2. I have two iscsi ports in each 2008 Server Node.

On Node A - I was able to grab a 7GB file and copy and paste it to my desktop in 12 seconds. It showed 500+ MBs a second.

However on Node B - I am only able to get 120 MBs a second and it takes 45 seconds to copy the same file. The two nodes are the exact same spec, processor, memory, etc...

Nic settings are the same. I used dells recomendation to set up iSCSI network accept for Jumbo frames and the Netsh commands.

I double checked the cards i didn't miss anything. The switches that are used in the high availability setup are the same.

I stopped and restarted the cluster and now Node one no longer copies at 500mbs a second. Now it only copies at 120mbs and takes 45 seconds to copy the same file.

What's weird is when it was copying at 500mbs a second I couln't see any iSCSI traffic on the TASK MANAGER NEtwork monitor tab and it said 0% under iSCSI Nics during the lightning fast copy . Now I can see the graph during the slower copy and it shows about 55%.

Can this be explained. I am using MPIO.
0
Comment
Question by:MEATBALLHERO
  • 5
  • 2
  • 2
  • +1
10 Comments
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
Comment Utility
Lots of things
 * Your benchmark is lying about when the file has actually been copied & flushed to disk. If you shut the computer off you might actually see that only 2-3 GB have actually been copied.
* Your benchmark includes a large number of tests, from uncached & cached reads, network transfer, and writes, yet you consider this nothing more than an iSCSI throughput test.

* The system cache is skewing the results.  That is why it is so fast at the beginning. It hits the ceiling after a few MB.


Set some baselines and do benching of both local reads/writes, then only test an iSCSI benchmark.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
If it's gigabit you can't go over 120MB/s no matter how fast your array may be.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
To summarize ... you are actually getting reasonable results, and test A is an invalid test because the O/S is lying about the file really being copied.   Nothing is wrong, you are getting what you should be getting.
0
 

Author Comment

by:MEATBALLHERO
Comment Utility
Wait...are you saying that even though it looks like I copied a 7GB file in 12 seconds, thats not true?

Are you saying that even though the window that shows the file copying closes, that the file is still copying in the back ground or something?
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
Yes, that is exactly what I am saying.   Do the math.  Let's ignore inefficiency and overhead.
7GB * 8 bits/byte / 12 seconds = 4.7 gigabits/sec

do you even HAVE a 4.7 gigabit connection, or do you have perhaps a 1Gigabit connection?

For that matter, I doubt you even have a disk drive that can read or write 7Gbytes in 12 seconds, let alone read the 7GB in the first 6 seconds, then write it at the other end in 6 seconds ... and, even then, we ignore the physics of the speed of light and the time it takes to move those electrons from point A to point B.

Bottom line, your benchmarks are giving you bad data. Based on the sustained numbers with the longer test, I see no reason for any concern.  Sure you can always do some tweaking, but don't expect anything more than +/- 20%
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 21

Expert Comment

by:eeRoot
Comment Utility
Can you verify the write speed of your desktop's hard drive?  And if the drive has built in cache?  You may get odd benchmark results if the drive has high speed memory to cache writes to disk, but then has to save that data to disk at regular disk write speeds.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
And what nics are in both systems
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
The NICs don't matter.  Benchmarking HAS to be wrong.   You are reading 7GB, transferring 7GB, and then writing 7 GB in all of 12 seconds.

That is roughly 2 GigaBYTES per second.  It doesn't matter what NIC you have.  You'd have to have infiniband, and some rather expensive RAID hardware to obtain this.  If you had such hardware you would know it.
0
 

Author Comment

by:MEATBALLHERO
Comment Utility
I have two San Servers that replicate. Both have a 16 port promise raid with 512MB of Cache. There are 16 SAS drives in each SAN that run at 15K RPM. this is where the 7GB file sits. I copied it to the desktop of my host which is 4 SAS raid 10 drives at 15K RPM. The drives all have cache enabled and so does the raid cards on all servers.

I do have a 10GB ethernet but that is connected SAN to SAN for replication and not SAN to Host.

Each San has 4 ISCSI Ports and My Host Has two but I put in the IP addresses of all 4 SAN Ports usining MPIO.
0
 
LVL 47

Expert Comment

by:dlethe
Comment Utility
It doesn't matter. No way can your hardware do 2Gbytes/sec.
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

We recently endured a series of broadcast storms that caused our ISP to shut us down for brief periods of time. After going through a multitude of tests, we determined that the issue was related to Intel NIC drivers on some new HP desktop computers …
This tutorial will give a an overview on how to deploy remote agents in Backup Exec 2012 to new servers. Click on the Backup Exec button in the upper left corner. From here, are global settings for the application such as connecting to a remote Back…
This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now