Solved

FILE COPY Saturates ISCSI Volume - Slow VM's

Posted on 2014-12-26
23
336 Views
Last Modified: 2016-11-23
Hi,
We have a 2 x node Dell R520 server 2012 Hyper-v setup.  Clustering not setup yet.
Each server has the following nic specs:
2 x on-board Broadcom’s
1 x 4 port Intel.

2 of the ports on the Intel nic are used strictly for IScsi traffic
The remaining nics are used in a LACP team.  VMQ has been disabled.  The LACP team is connected to 2 x Stacked 3750 Cisco switches.  All appropriated Trunk ports, speed, media, vlans and MTU have been applied to the switch config and ensure ISCSI traffic is restricted to the VLANS.

A V-switch has been setup with standard networks connected to it (Management, Migration and Cluster)
All networks are on separate subnets and Vlans.

ISCSI is provided via an ISCSI Lun on a Synology  RS10613xs+ Cluster with 15k 900gb drives with MPIO enabled and 2 nics on each node dedicated to ISCSI vlans.

ISCSI targets are successfully added to each Hyper-v node and MPIO enabled.  Have verified I can see the 2 sessions for each target from each node.  Jumbo frames enabled on each nic on the Hyper-v nodes along with the SAN and the switches.  Testing confirmed.

Initial testing is ok, performance is good but can be better.

Our problem is that the moment we attempt to read or write anything to the ISCSI volumes (STD file copy), all VMs grind to a halt.  Pausing or stopping the file copy to or from the ISCSI volumes restores normal connectivity on the VM’s.

Examining the Synology cluster we can see it’s under no stress and happily serving the LUNS.
Confirmed the traffic for both the file copy and VM traffic is only occurring on the ISCSI networks / VLANS across the hyper-v nodes and the SAN.

The VM’s that are currently running are under no stress or load.
A simple copy from the ISCSI volume and to the volume gives an average of 600MB/S (while peaking to 800MB/s at times)

We are at a loss as to where the congestion / bottleneck is coming from or occurring.

Any help is much appreciated.
Thanks
Sam
0
Comment
Question by:Sam Newbie
  • 12
  • 4
  • 4
  • +1
23 Comments
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 40519066
600MB/s = 6Gbits/sec

What speed is your NIC and what speed does your switch do?
0
 

Author Comment

by:Sam Newbie
ID: 40519145
Hi Gerald,
All Nics are 1GBs.  Switches are STD Gigabit Cisco Switches setup with Port Groups (LACP) and trunked to the team.
As I said, speed of file transfer is acceptable, however my concern is a normal file copy to and/or from the ISCSI volumes where the VM's sit saturates the ISCSI networks and renders the virtual machines to be unusable.

Thanks
Sam
0
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 40519152
What happens if you do the copy to a different volume than the one your VMs are on? Best would be to try from a host not running your VMs. If it's still bad, the problem in in your Synology or network.

I would check to confirm that your jumbo frames are actually working properly, as a large file copy would use large frames but standard IO for a VM might not. Off the top of my head "ping aaa.bob.ccc.did -f -l 8950".

It shouldn't be causing the issue, but I don't believe that your switches are really suitable to iSCSI traffic because their small buffers can't handle microbursts. They are more suitable to workstations than servers and storage. Catalyst 4900 series are actually designed to handle iSCSI traffic.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 40519214
If you only have 1GbE NICs then your copy is using every port possible, are you sure you only have 2 ports set for iSCSI Traffic? With only 2 ports I would expect max speed of 150MB/s
0
 

Author Comment

by:Sam Newbie
ID: 40519246
Hi Kevin,
File copy speed is not the problem though.  I can copy to and from the Synology on either the volume hosting the VM's or other volume with great speeds.  My problem is that these copies render the VM to be really slow (unusable).  Connectivity to the VM's is still maintained however extremely slow.

Jumbo frame works perfectly from the switch to all to Synology on all ports, to the ISCSI network on the servers and back to the switch interface.

I agree with you about the switches, however we are talking about an environment which has only 2 test VM's, doing absolutely nothing, no other users on the network and no other activity.
0
 

Author Comment

by:Sam Newbie
ID: 40519248
Hi Gerald.
Definitely.  Setup is as follows :
Servers
2 x Intel 1gb Ports.  Each Port setup as follows :
Port 1: Vlan 3 member on switch = 192.168.3.x - No default gateway
Port 2 : Vlan 4 member on switch= 192.168.4.x - No default gateway
MPIO enabled
Jumbo frames enabled

Switches
System MTU set to Jumbo Frames
On each interface we have the following :
Switchport mode access
switchport access vlan (either 3 or 4)
speed1000
duplex full

On the Synology
2 x ports dedicated to ISCSI
Port 3 and Port 4
Each enabled with jumbo frames and assigned an ip address relevant to the respective ISCSI Vlan
0
 

Author Comment

by:Sam Newbie
ID: 40519278
Sorry, now that I read what I typed in my initial post I see where you are coming from.

Copying from ISCSI volume to the same volume  = is over 600MB/s (sustained)
copying from ISCSI volume to hyper-v host = 170 MS/s sustained
copying from hyper-v host to ISCSI volume = 170 MS/s sustained

Just to confirm, I am prefectly happy with copy speeds, just not with it saturating the ISCSI network.
0
 

Author Comment

by:Sam Newbie
ID: 40519286
One event I am getting which I don't know if its related is the following :

MAC conflict: A port on the virtual switch has the same MAC as one of the underlying team members on Team Nic Microsoft Network Adapter Multiplexor Driver.

I manually went and assigned the reported members a different Mac ADD however the error is still reported.

Doing a get-netadapter | sort macaddress
Does not show any duplicated (Unless I am missing something?)

get-netadapter | sort macaddress 

Open in new window

00-15-5D-F9-8E-00
00-15-5D-F9-8E-01
A0-36-9F-4B-E7-44
A0-36-9F-4B-E7-45
A0-36-9F-4B-E7-46
A0-36-9F-4B-E7-47
B0-83-FE-C9-B4-2C
B0-83-FE-C9-B4-2D
B0-83-FE-C9-B4-2F
B0-83-FE-C9-B4-30
B0-83-FE-C9-B4-32
0
 

Author Comment

by:Sam Newbie
ID: 40519543
Further to this, at 11pm with no users connected to the environment, no load on any VM.  Running Microsoft Exchange Jetstress brings the ISCSI volume to a crawl.  All VM's becomes non-responsive and the Hyper-v  hosts begin reporting lots of ISCSIPrt error.

Anyone have any ideas?
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 40519555
it really sounds like you have some kind of network error. have you tried turning it ALL off and bring each bit on line one piece a a time to isolate the two different network errors you have reported so far?
0
 

Author Comment

by:Sam Newbie
ID: 40519601
yeah, once everything has been restarted or once the Jetstress or the copy operation stops everything returns to normal.
Shutting VMs, servers, san and switches and starting each up one by one still produces the same problem.
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 27

Expert Comment

by:Steve
ID: 40521555
apologies if I repeat anything that's already been mentioned, but I notice you have 2 x NICs on each node & 2 X NICs on the SAN for iSCSI?

Surely that means the servers can communicate at a total of 4 x 1Gbit while the SAN is saturating it's 2 x 1Gbit bandwidth to keep up with both nodes?
You should normally design a SAN with at least 2 x the bandwidth available to each individual node (or 4 x in heavy load environments)

I also notice that you have the 2 x NICs on each host on different VLANs & subnets....
Generally, I believe it is preferred to have the NICs on the same network so the system can work out how to spread the traffic across both NICs.

This means the SAN has a single IP which is shared by as many NICs as you can give it, and the hosts have multiple NICS able to see the SANs IP. this allows MPIO to work out how each NIC can reach the SAN and use it's bandwidth efficiently.
0
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 40521682
My question is, when you are stressing the system by putting an IO load on one of the volumes, is another volume being accessed by a different host having issues? It isn't clear from the posts what happens in this case.

The jumbo ping should be tested from the hosts to the storage. Don't test from the switch.

Can you post the configuration of the ports on your switch, and the output from "show int" for the ports used by the Synology and hosts?
0
 

Author Comment

by:Sam Newbie
ID: 40523027
Hi totallytonto,
Correct, 2 nics on each node and 2 nics on each NAS.  The NAS is setup in Active/Passive not Active/Active.

No, Clients do not use the ISCSI network. This is a completely virtualised environment.  All VM's are on a management network.  VM's are then stored on the ISCSI volume.  Any traffic to Storage (VM) is done fir ISCSI3x and ISCSI4x dedicated networks.

Client traffic is segregated to a separate management vlan.
We are of the understanding that MPIO setup recommendation is to have 2 different networks on both ends segregated in order to achieve maximum performance.  Putting them all on the same network would saturate the network even more with broadcast traffic?
0
 

Author Comment

by:Sam Newbie
ID: 40523067
kevinhsieh

If I do a copy on one host to another volume, then commence another copy on another volume from the same host.  Results are the same.  Network is saturated.

If I do a copy or test on one host to a volume, then go to another host and perform another copy to another volume.  Still same results.

At the time of copying, I can also see lots of ISCSI timeout errors  in the host event viewer.

At the time of copying, I have set the load-interval on the switch interface to 30 and if I do a show interface .. and show process cpu shows everything as normal.

Jumbo frame tests have been done from end to end.  From server to san, san to server, server to switch, switch to san, switch to server, etc..

Switch configs are : (A little long)

interface Port-channel1
 description Server1 Team LACP
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
!
interface Port-channel2
 description Server2 Team LACP
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk

interface GigabitEthernet1/0/2
 description SAN1 Port2 Management
 switchport access vlan 2
 switchport mode access
 speed 1000
 duplex full
!
interface GigabitEthernet1/0/3
 description SAN2 Port3 ISCSI3x
 switchport access vlan 3
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet1/0/4
 description SAN1 Port4 ISCSI4x
 switchport access vlan 4
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast

interface GigabitEthernet1/0/6
 description HVNode1 Team Onbrd1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 1 mode active
!
interface GigabitEthernet1/0/7
 description HVNode2 Team Onbrd2
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 2 mode active
!
interface GigabitEthernet1/0/8
 description HVNode1 ISCSI4x IntelP4
 switchport access vlan 4
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet1/0/9
 description HVNode2 ISCSI3x IntelP3
 switchport access vlan 3
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet1/0/10
 description HVNode1 Team IntelP2
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 1 mode active
!
interface GigabitEthernet1/0/11
 description HVNode2 Team IntelP1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 2 mode active
!
!
interface GigabitEthernet2/0/2
 description SAN2 Port2 Management
 switchport access vlan 2
 switchport mode access
 speed 1000
 duplex full
!
interface GigabitEthernet2/0/3
 description SAN1 Port3 ISCSI3x
 switchport access vlan 3
 switchport mode access
 speed 1000
 duplex full
!
interface GigabitEthernet2/0/4
 description SAN2 Port4 ISCSI4x
 switchport access vlan 4
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
!
!
interface GigabitEthernet2/0/6
 description HVNode2 Team Onbrd1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 2 mode active
!
interface GigabitEthernet2/0/7
 description HVNode1 Team Onbrd2
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 1 mode active
!
interface GigabitEthernet2/0/8
 description HVNode2 ISCSI4x Intel4
 switchport access vlan 4
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet2/0/9
 description HVNode1 ISCSI3x IntelP3
 switchport access vlan 3
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
!
interface GigabitEthernet2/0/10
 description HVNode2 IntelP2 Team
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 2 mode active
!
interface GigabitEthernet2/0/11
 description HVNode1 IntelP1 TeamLACP
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,6,7
 switchport mode trunk
 speed 1000
 duplex full
 channel-group 1 mode active



================================

Switch1#sh int gi 1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e603 (bia 0012.00dd.e603)
  Description: SAN2 Port3 ISCSI3x
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     3638 packets input, 3887662 bytes, 0 no buffer
     Received 3638 broadcasts (3637 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 3637 multicast, 0 pause input
     0 input packets with dribble condition detected
     41863 packets output, 3254007 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#

Switch1#sh int GigabitEthernet1/0/4
GigabitEthernet1/0/4 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e604 (bia 0012.00dd.e604)
  Description: SAN1 Port4 ISCSI4x
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     369539 packets input, 496796350 bytes, 0 no buffer
     Received 1453 broadcasts (87 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 87 multicast, 0 pause input
     0 input packets with dribble condition detected
     122867 packets output, 31586715 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 1/0/6
GigabitEthernet1/0/6 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e606 (bia 0012.00dd.e606)
  Description: HVNode1 Team Onbrd1
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 17:27:29, output 17:27:02, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     341 packets input, 113882 bytes, 0 no buffer
     Received 200 broadcasts (161 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 161 multicast, 0 pause input
     0 input packets with dribble condition detected
     9575 packets output, 1504958 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 1/0/7
GigabitEthernet1/0/7 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e607 (bia 0012.00dd.e607)
  Description: HVNode2 Team Onbrd2
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 17:27:26, output 17:27:11, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     15494 packets input, 2176365 bytes, 0 no buffer
     Received 659 broadcasts (320 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 320 multicast, 0 pause input
     0 input packets with dribble condition detected
     6836 packets output, 1045211 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 1/0/8
GigabitEthernet1/0/8 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e608 (bia 0012.00dd.e608)
  Description: HVNode1 ISCSI4x IntelP4
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:28:21, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     211 packets input, 19396 bytes, 0 no buffer
     Received 197 broadcasts (151 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 151 multicast, 0 pause input
     0 input packets with dribble condition detected
     2141 packets output, 310616 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#

Switch1#sh int gi 1/0/9
GigabitEthernet1/0/9 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e609 (bia 0012.00dd.e609)
  Description: HVNode2 ISCSI3x IntelP3
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:28:13, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     63622 packets input, 22652819 bytes, 0 no buffer
     Received 732 broadcasts (446 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 446 multicast, 0 pause input
     0 input packets with dribble condition detected
     131275 packets output, 485983874 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 1/0/10
GigabitEthernet1/0/10 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e60a (bia 0012.00dd.e60a)
  Description: HVNode1 Team IntelP2
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 17:29:45, output 17:29:20, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     235 packets input, 67609 bytes, 0 no buffer
     Received 130 broadcasts (130 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 130 multicast, 0 pause input
     0 input packets with dribble condition detected
     2132 packets output, 266070 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#

Switch1#sh int gi 1/0/11
GigabitEthernet1/0/11 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 0012.00dd.e60b (bia 0012.00dd.e60b)
  Description: HVNode2 Team IntelP1
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 17:29:59, output 17:29:44, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     9768 packets input, 1639498 bytes, 0 no buffer
     Received 585 broadcasts (465 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 465 multicast, 0 pause input
     0 input packets with dribble condition detected
     2903 packets output, 320253 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#

Switch1#sh int gi 2/0/3
GigabitEthernet2/0/3 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001a.a212.8683 (bia 001a.a212.8683)
  Description: SAN1 Port3 ISCSI3x
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:43, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     131228 packets input, 485924469 bytes, 0 no buffer
     Received 1422 broadcasts (93 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 93 multicast, 0 pause input
     0 input packets with dribble condition detected
     107042 packets output, 29544243 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 2/0/4
GigabitEthernet2/0/4 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001a.a212.8684 (bia 001a.a212.8684)
  Description: SAN2 Port4 ISCSI4x
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:04, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     3666 packets input, 3919634 bytes, 0 no buffer
     Received 3666 broadcasts (3665 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 3665 multicast, 0 pause input
     0 input packets with dribble condition detected
     42253 packets output, 3277487 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 2/0/6
GigabitEthernet2/0/6 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 001a.a212.8686 (bia 001a.a212.8686)
  Description: HVNode2 Team Onbrd1
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:34:37, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     14415 packets input, 2143783 bytes, 0 no buffer
     Received 733 broadcasts (397 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 397 multicast, 0 pause input
     0 input packets with dribble condition detected
     32373 packets output, 2888768 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#

Switch1#sh int gi 2/0/7
GigabitEthernet2/0/7 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 001a.a212.8687 (bia 001a.a212.8687)
  Description: HVNode1 Team Onbrd2
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:36:22, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     266 packets input, 85437 bytes, 0 no buffer
     Received 155 broadcasts (114 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 114 multicast, 0 pause input
     0 input packets with dribble condition detected
     10990 packets output, 1030117 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#





Switch1#sh int gi 2/0/8
GigabitEthernet2/0/8 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 001a.a212.8688 (bia 001a.a212.8688)
  Description: HVNode2 ISCSI4x Intel4
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:35:26, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     79496 packets input, 24721929 bytes, 0 no buffer
     Received 677 broadcasts (403 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 403 multicast, 0 pause input
     0 input packets with dribble condition detected
     369624 packets output, 496855945 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 2/0/9
GigabitEthernet2/0/9 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 001a.a212.8689 (bia 001a.a212.8689)
  Description: HVNode1 ISCSI3x IntelP3
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:37:07, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     157 packets input, 14890 bytes, 0 no buffer
     Received 143 broadcasts (109 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 109 multicast, 0 pause input
     0 input packets with dribble condition detected
     2122 packets output, 315240 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 2/0/10
GigabitEthernet2/0/10 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 001a.a212.868a (bia 001a.a212.868a)
  Description: HVNode2 IntelP2 Team
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:36:21, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     10093 packets input, 1696162 bytes, 0 no buffer
     Received 590 broadcasts (439 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 439 multicast, 0 pause input
     0 input packets with dribble condition detected
     2572 packets output, 391270 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#


Switch1#sh int gi 2/0/11
GigabitEthernet2/0/11 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 001a.a212.868b (bia 001a.a212.868b)
  Description: HVNode1 IntelP1 TeamLACP
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 17:38:11, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     227 packets input, 65626 bytes, 0 no buffer
     Received 129 broadcasts (102 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 102 multicast, 0 pause input
     0 input packets with dribble condition detected
     3639 packets output, 492441 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Switch1#
0
 
LVL 42

Expert Comment

by:kevinhsieh
ID: 40524795
I don't see flow control enabled, but that shouldn't cause the type of issues you are seeing. I don't see any errors on your switch, though a lot of the interfaces you posted are disconnected. I haven't cross checked to be sure that the iSCSI interfaces are the ones that are connected.

iSCSI is generally not supported with NIC teaming, so be sure you aren't doing that.

Have you checked with Synology? The problem seems to be that one type of IO workload stomps all over everything else. NOrmally that's a problem with the storage, not the network.
0
 
LVL 27

Expert Comment

by:Steve
ID: 40531179
not sure which port is which, but agree with kevin, make sure you aren't using teaming on iSCSI ports as it doesn't like it. MPIO is way better that teaming so you don't need both.
0
 

Author Comment

by:Sam Newbie
ID: 40532653
Hi guys,
Just reporting back.  Definately not using teaming on the ISCSI network.  As per my post there are 2 ISCSI networks on 2 separate Vlans with no default gateways setup in MPIO.  Have confirmed that 2 sessions per network exist from each node.

After further testing, we have concluded that the network, setup and config is fine along with Hyper-v config.  We are adding more hardware to the Synology side.  We only had 4 hard drives (900gb 10k SAS ) per NAS setup in raid 5.  Will be adding another 4 drives and test further.

Problem still unresolved, pending more parts to arrive and further testing. Leave this open or close it?

Appreciate all your help.
0
 
LVL 27

Expert Comment

by:Steve
ID: 40533128
can you ping the iSCSI IPs? have you tried leaving a few pings running from the hosts while you conduct your tests to see if traffic appears to be affected?


After further testing, we have concluded that the network, setup and config is fine along with Hyper-v config.  We are adding more hardware to the Synology side.
Is there a reason you have now reversed your original conclusion that the SAN is fine? Has testing revealed an issue?
Examining the Synology cluster we can see it’s under no stress and happily serving the LUNS.

I definitely recommend some testing to see how the SAN is coping to be sure, but it would certainly rule out most of our advice if the SAN turned out to be the issue.
0
 
LVL 16

Assisted Solution

by:Gerald Connolly
Gerald Connolly earned 250 total points
ID: 40533255
RAID-5 isnt recommended for large disk any more due to the Risk exposure in the long rebuild times, go for RAID-6 or RAID10
0
 

Author Comment

by:Sam Newbie
ID: 40534917
hi Guys,
During our testing leaving constant pings across both networks and latency was minimal if any.
We also noticed IO wait times on the NAS.  

Network testing was great on all occasions which also included replacing equipment.
Will definitely go for the Raid 6 with 8 drives per NAS.

Really appreciate your help guys.
0
 
LVL 42

Accepted Solution

by:
kevinhsieh earned 250 total points
ID: 40534955
With 10K drives, RAID5 isn't so bad. RAID6 will have really slow writes, but nice reads. RAID10 will have good writes and pretty good reads.
0
 

Author Closing Comment

by:Sam Newbie
ID: 40545570
Thanks guys, really appreciate your support.  Have changed storage config a little and still performing testing.
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

The article will show you how you can maintain a simple logfile of all Startup and Shutdown events on Windows servers and desktops with PowerShell. The script can be easily adapted into doing more like gracefully silencing/updating your monitoring s…
What to do when Windows Update is not working correctly? What tools can I use to detect the cause of the malfunction problem? What does this numeric error code mean? These and other questions that you have been asking in the past are answered here (…
In this Micro Tutorial viewers will learn how to restore single file or folder from Bare Metal backup image of their system. Tutorial shows how to restore files and folders from system backup. Often it is not needed to restore entire system when onl…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now