Solved

switching on switch

Posted on 2011-03-21
26
499 Views
Last Modified: 2013-05-04
we have foundry switches and router. question about layer 2 or 3 routing.

if a switch is connected to a router and said switch has a server and an iSCSI SAN connected to it, but in different subnets and the server has a volume on the iSCSI SAN connected to it...

will the path that the data takes to be stored on the SAN go from the server to the switch, up to the router, back down to the same switch and down to the san? Or will the switch handle this handoff even though the connections are in different subnets and vlans?
0
Comment
Question by:MrVault
  • 8
  • 8
  • 7
  • +1
26 Comments
 
LVL 55

Assisted Solution

by:andyalder
andyalder earned 167 total points
Comment Utility
If the server and storage are in different subnets then all traffic would have to go through the router, but why put them on different subnets? Additional NICs are cheap, best to have a pair of them or at least a dual port one dedicated to iSCSI.
0
 

Author Comment

by:MrVault
Comment Utility
the reason for putting on separate subnets and vlans is to reduce the overhead broadcasting, traffic security. I'm not a networking person but every best practice article I've ready on building a an iSCSI traffic network recommends segregating the traffic at least virtually if not physically from the rest of the traffic. I guess I was hoping that a layer 2 switch would be able to switch the traffic locally if both devices were plugged into it instead of having to go up a hop.
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
You put your servers iSCSI HBA on a different subnet than the storage it talks to to reduce broadcasting?

I think you need to think that one out again; you can certainly use vlans to segregate traffic (you don't want your LAN and SAN on the same subnet) but you don't use them to separate things that continually talk to each other.  

Say you only had one NIC per server but it supported vlan tagging, you could split that NIC into four logical ones so pretend you've got 4 NICs in it for the design phase and also pretend your vlan capable switch is also four physical boxes. You wouldn't cross connect them and add a router would you?
0
 
LVL 9

Accepted Solution

by:
Red-King earned 167 total points
Comment Utility
What you want is to have your SAN and the Server NIC for iSCSI on the same subnet and VLAN.
The management port of your SAN and the Server NIC for LAN traffic will then be in a different subnet and VLAN.
You configure the interfaces on the switch to be in particular VLANs. You then plug the SAN and Server NICs in appropriately.
In this way you don't send traffic to the router as there's no need for the LAN ports to access the iSCSI ports and vice versa. If there's a firewall built into the router you can block traffic between both subnets for security.
0
 
LVL 4

Assisted Solution

by:m_walker
m_walker earned 166 total points
Comment Utility
To add to Red-Kings post...
If you have your SAN VLAN and do not apply an IP Address to it it will remain at layer 2 only and no one anywhere should be able to route into it.  A vlan only needs an IP Address if you want to be able to route to it.

If budget allows, I would put the iScsi on its own switch, not only does this reduce the broadcast from other computers as with the vlan, but it will also ensure the switch is 100% for san access.

I fully agree with others in that you should have at least 1 nic of the iSCSI/HBA and one for user access, as outlined already, if you have 1 nic on the server for both iSCSI and user access then the NIC will be dealing with all broadcasts from the user vlan already, so routing will be adding to the overhead and not helping.

My advice (as others have said)
1. SAN on its own switch layer 2 only for best performance.
2. SAN on its on VLAN with server having 2 nics, one in SAN VLAN one in user VLAN.
Then QOS the SAN traffic
3. SAN and users all in the one vlan, but only of option 1 and 2 cant happen.
0
 

Author Comment

by:MrVault
Comment Utility
Thanks people. I totally agree about separating them out with separate NICs, subnets, and VLANs.

The thing was I thought I remembered someone once telling me that if I couldn't afford separate switches for iSCSI and LAN traffic, that I could just VLAN out a single switch and use ACLs to keep the traffic separate. My question though is if the data packets would actually go from server iscsi NIC to switchport on iSCSI VLAN, to router, back to switch, out SAN iSCSI switchport and down to the SAN or if it would never go up to the router.

If we are able to purchase separate iSCSI switches, we will still need to give them an IP because we need to replicate the SAN to another SAN that's is only accessible down the path that goes through the router.
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
VLANs will keep them seperate if you use one subnet for the LAN and one for the SAN, so ACLs shouldn't come into it as traffic from one address on a subnet doesn't go through a router to get to another IP address on the same subnet. Only the remote replication iSCSI traffic will go through the router as your remote SAN is on a different subnet.
0
 

Author Comment

by:MrVault
Comment Utility
so here's our situation.

we have one router on on side of the campus. on the other side we have a switch with a SAN and a server attached to the switch. we want to segregate the traffic with VLANs but our hope was that traffic wouldn't have to go all the way back to the router when the server is writing data to it's iSCSI volume. I realize one option is to purchase a separate switch for iSCSI that the server's iSCSI NIC would connect to and the SAN would connect to. I was just hoping that we could avoid buying another switch and still separate out the traffic with VLANs.

thanks!
0
 
LVL 55

Assisted Solution

by:andyalder
andyalder earned 167 total points
Comment Utility
If the servers iSCSI is on the same subnet / VLAN as the storage the traffic will stay local, if it's on a different subnet it has to go through a router. You could always treat your cross-campus link like a WAN link and put a router each side of it although that would require a bit of network modification.
0
 
LVL 9

Assisted Solution

by:Red-King
Red-King earned 167 total points
Comment Utility
When you section off switchports into a VLAN you are essentially breaking up your physical switch into smaller logical switches, kinda like partitioning a hard drive.
If you create an iSCSI VLAN and a Data VLAN traffic will have to go to the router to go between the VLANs
If the server has 2 NICs then configuring one for the same subnet as the SANs iSCSI port will allow you to segregate the traffic with VLANs as you would like to.
The iSCSI traffic will only then go through the router if it is destined for another subnet

For example, If somebody accesses a fileshare on the SAN through the server then the following will happen
Request is received by the Server on the Data VLAN
Server sends request for data to the SAN via the iSCSI VLAN

SAN replies to request to the server via iSCSI VLAN

Server replies to the users request (with the information from the SAN) via the Data VLAN

In this way there is no need for the iSCSI traffic to go to the router. It will only go through the router if there is direct SAN to SAN replication and the second SAN is on a different network.

Of course, if the server only has one NIC then you would have to consider what andyalder said earlier about vlan tagging on the NIC.


0
 

Author Comment

by:MrVault
Comment Utility
thanks everyone. I think I've got it now. just one last question - what advantage does vlan tagging have on a single NIC? the nic's utilization would be the same, no?
0
 
LVL 4

Assisted Solution

by:m_walker
m_walker earned 166 total points
Comment Utility
When you are talking about a nic in a server, it may or may not see the vlan tags.
In most cases the switch port that has the nic connected will be set to "untag" when the data goes to the Server and tag the data when it enters the switch.  If you need the data tagged when going to the server then the vlan tags will add a few bytes to each packet, so the mtu may need tweaking, so you are right in no real advantage.

That said, if a server has only 1 nic and (for example) you are running vmware, you may what differnet vm servers in different vlans.  So you would pass the vlan tags to the vmserver so it can have virtual servers in different vlans. thus avoiding routing and easier security and better performance.

Short answer: VLAN taging to the server is good when you need many vlans to appear on the server.

 
0
 

Author Comment

by:MrVault
Comment Utility
thanks.
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 55

Expert Comment

by:andyalder
Comment Utility
It would also allow you to run jumbo frames for iSCSI and normal frames for data wouldn't it? I thought all server grade NICs supported vlan tagging but yeah, it could have a cheap one in it.
0
 
LVL 4

Expert Comment

by:m_walker
Comment Utility
andyalder: While the nic could do it, the os/nic drivers nead to deal with it.  Jumbo frames can help, but be sure the jubmo frames are supported all the way from server to server.  If the switch needs to fragment then you could take a very big hit.

eg: All the way (mtu size for example only)
Svr1 9000 MTU -> Switch 1 9000 MTU -> Switch 2 9000 MTU -> Svr 2 9000 MTU
this will work well as the data to overhead is very good.

Bad example
Svr1 9000 MTU -> Switch 1 9000 MTU -> Switch 2 1500 MTU -> Svr 2 9000 MTU
In this example, when the 9000 byte packet gets to switch 2, switch 2 (depending on setup and packet flags) will
a) Just drop the packet (very very bad) Old gear can do this.
b) Send an ICMP "need to fragement, but dont fragment flag set
   Will happen so ensure ICMP packets are allowed on any server level firewall.
c) Fragment the packet.
 Switch 2 will take the 9000 byte packet, then breakup into 1500 byte packets (with all the header info), this will take switch cpu time and be slow.  Then Server 2 will need to take in all packets and rebuild prior to passing up the OSI Levels to the application.  2nd level delay.

You dont need vlans for jumbo frames, you can do them on a single nic, but your switch may support per vlan MTU settings so correct placement of device into the correct vlan would be better.
0
 

Author Comment

by:MrVault
Comment Utility
I wonder if the jumbo frame topic is why we're having issues right now.

Right now our servers have 2 onboard NICs which in a flat network provide the LAN and iSCSI traffic (single IP per NIC, one VLAN).

on the primary NIC we have all the windows protocols necessary enabled, but the NIC is configured for iSCSI (jumbo frames, MTU 9000, flow control, etc). The other NIC has same settings, but no protocols, just IPv4 and no gateway. Every once in a while some random server goes blank when we RDP to it. There's no way to fix it except rebooting. The server is still working just fine in terms of customer connections, iSCSI load, etc. But we can't get in. I wonder if because we're RDP-ing over a NIC that has MTU 9000 (and the switch is MTU 10222) that it randomly loses control of the main connection.
0
 
LVL 4

Expert Comment

by:m_walker
Comment Utility
As long as the switchs have bigger MTUs then the servers it will be fine.  The MTU Path should be the same (or bigger) then the mtu each end device is using.  When you vlan the switch will add a few bytes to the packet for the vlan tag.  

If you can use and understand wireshark (or some packet sniffer) use that to see if DUP packets are seen or ICMP need to fragment are seen.  
In a stock network I would have 1500MTU on the clients and 1546 for the switch MTU.  the extra 46 are to support QOS, VLAN and other tags added to the packet for our network setup.  In this model, the full client 1500 byte packet will make it from host to host.

You may find the jumbo MTU is a little too big under load, so you could try making it a little lower, but ensure each end can deal with the jumbo size.
 
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
I wouldn't actually bother with jumbo frames, I just introduced it as an example of how you might need VLAN tagging at the server NIC.

I don't think you're right with "You dont need vlans for jumbo frames, you can do them on a single nic," though, at least not with a server with a single NIC because you must not run more than one frame size on any particular vlan, so you mustn't run normal LAN sized frames plus jumbos on the same vlan.
0
 
LVL 4

Expert Comment

by:m_walker
Comment Utility
Comment accepted andyalder.  If you were to use jumbo frames for both iSCSI and client access, this could be bad.  My comment was more about if you have a nic and it is only used for the iSCSI, then you dont need to vlan it, but can still use jumbo frames.
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
Agreed, we're talking at cross-purposses to some extent, it would be a lot better to have separate NICs and switches dedicated to LAN and SAN. I come from a fibre channel background so there's none of this trying to run two protocols on one network.
0
 

Author Comment

by:MrVault
Comment Utility
what do you guys think about iSCSI HBAs verses just separate regular PCIe NICs? Our servers have 2 onboard NICs and we want to give those to HA LAN traffic, so we're adding extra NICs for iSCSI. But vendor is saying we don't need iSCSI HBA just regular NICs. right now we don't boot from SAN, but our CPU utilization does get high so we're thinking it'd be good to offload the iSCSI overhead on the CPU.

Thoughts? These are Dell poweredge 2950, r510, and r710 servers.
0
 
LVL 9

Expert Comment

by:Red-King
Comment Utility
When I hear HBA I generally think Fibre so maybe that is what you're vendor is thinking.
HBA can really be applied to anytype of adaptor though I guess. All the iSCSI connections I've worked with so far have been using standard ethernet NICs. They're perfectly acceptable as far as I'm concerned.
0
 

Author Comment

by:MrVault
Comment Utility
yeah, there are iSCSI hba and fibre hba. and yes i could use a regular nic too, but there has to be some sort of advantage between a regular nic, and toe nic and a iscsi hba. thanks though.
0
 
LVL 4

Expert Comment

by:m_walker
Comment Utility
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
Whilst iSCSI and TCP offload do save CPU resource at $500 per port it's generally cheaper to buy faster CPUs.
0
 
LVL 4

Expert Comment

by:m_walker
Comment Utility
I am only just playing with this now, so keen to see how it pans out.
I setup a fedora 14 box with ext3.  Then on that setup the free iSCSI server (unsing flat files on the ext3).  I kinda saw this as a worst case setup.  I made the iSCSI connection from the free esxi vm server and setup 2003 srv on the iSCSI LUN.  the nic on the VM Server went to a cisco 3750 switch and the linux server was on that same switch (no other devices) as an out of band connection.

The 2003 "vm nic" went onto the live network.  I was getting between 850-900 Mbps transfer from my notebook to the 2003 server then out the other nic to the linux iSCSI lun.  Since each nic was a 1Gbps, I thought this was not too bad.

That said, a direct write on the linux server to create the flat file wrote at a rate of 3Gbps for a 100Gig file.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
Don’t let your business fall victim to the coming apocalypse – use our Survival Guide for the Fax Apocalypse to identify the risks and signs of zombie fax activities at your business.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now