Solved

NetApp SAN switch network design question

Posted on 2014-04-13
28
903 Views
Last Modified: 2014-04-26
I have a NetApp, a switch and two ESXi servers in my rack. The switch in the rack is connected to the core switch for our LAN via ether channel. It looks like this:



[core switch]---------------------------[storage switch]--------------[NetApp]--------[ESXi 1 & ESXi2]  

My question: is this a typical design?  Do most environments have a separate storage switch sitting between their storage and ESXi servers or do they just connect everything to the core of the LAN?

What is the best practice approach?

Thanks
0
Comment
Question by:wayy2be
  • 12
  • 9
  • 7
28 Comments
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 250 total points
Comment Utility
It could be, but we install seperate storage switches, usually two, in a STACK, and then comnnect at least two paths from every switch to each ESXi server, using Multipath.

(not teamed or ether channel connections, e.g. trunks)

Best Practice is Multipath, and enable jumbo frames make help.

HOW TO: Add an iSCSI Software Adaptor and Create an iSCSI Multipath Network in VMware vSphere Hypervisor ESXi 5.0

HOW TO: Enable Jumbo Frames on a VMware vSphere Hypervisor (ESXi 5.0) host server using the VMware vSphere Client
0
 
LVL 10

Expert Comment

by:Schuyler Dorsey
Comment Utility
As Andrew mentioned, you certainly should have two storage switches stacked in this scenario as you have created a single point of failure. If that one storage switch goes down, all of your VMs will go down.
0
 

Author Comment

by:wayy2be
Comment Utility
Absolutely I agree and that is what I am planning to do in the next few weeks. Budget was tight last year so we could only afford one switch.  So how does the storage switch stack get connected to the core LAN switch?  What is the best way?
0
 
LVL 10

Expert Comment

by:Schuyler Dorsey
Comment Utility
Typically a port-aggregation. You could do two ports (one from each storage switch) to the core switch.
0
 

Author Comment

by:wayy2be
Comment Utility
I did have a vendor tell me to forget the storage switch and connect straight to the core with the ESXi and NetApp. However I did not feel that this was correct. What do you think of that?  The current storage switch is a Cisco 3750. Is that a good switch for this purpose or overkill?

Thanks for the replies guys
0
 
LVL 10

Expert Comment

by:Schuyler Dorsey
Comment Utility
Actually.. that switch is BELOW what I normal spec. Cisco 3750's often have difficulty keeping up with storage due to the low cache buffer sizes. It will REALLY affect your switch is your storage environment does a burst.

If we are specc'ing Cisco, we do a MINIMUM of Cisco 3850. If we do HP, we spec HP 3800s.

As far as connecting directly to the core, I see that a lot at small businesses. So I suppose it would depend on the size of your network/business. It is common practice to use a separate switch for storage.
0
 

Author Comment

by:wayy2be
Comment Utility
We have around 200 clients and 25 vm's.  What would you suggest in Brocade?  If I bought Brocade would they place nice with our Cisco core, which are also 3750's?
0
 
LVL 10

Expert Comment

by:Schuyler Dorsey
Comment Utility
I can't speak on the Brocade side. I have only worked with their fiber switches. Those are okay but I'm not crazy about the mgmt side of them.

It should play nicely as long as you are not using Cisco proprietary protocols such as EIGRP.  In this case, you would have to add another routing protocol to your network such as OSPF.

Any reason you mentioned Brocade? The specific HP and Cisco switches I mentioned would do wonderfully. The HP one would actually be a better performer than the Cisco, has lifetime warranty, lifetime updates and would be much cheaper.
0
 

Author Comment

by:wayy2be
Comment Utility
No reason, just looking at all the options. Would the HP play nice with the Cisco core?
0
 
LVL 10

Expert Comment

by:Schuyler Dorsey
Comment Utility
Same with the Brocade.. you only have to consider if you are running any proprietary Cisco protocols like EIGRP. HP is an open standards based switch so it will support RIP and OSPF. Cisco DOES allow you to inject EIGRP routes into OSPF if needed.

But I have several environments with both HP and Cisco running without issue.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
We use Brocade or Dell Power Connect 7024 Storage switches in a stack, just for storage.

These are not connected to any other networking, it's the storage network is isolated and runs Jumbo Frames.
0
 

Author Comment

by:wayy2be
Comment Utility
Andrew:  So how are you connected to your LAN?
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
We have a seperate completely dedicated 2 to 4 storage switches, which ONLY NFS and iSCSI traffic passes from ESXi servers to SAN.

These are Storage Switches purchased for the purpose of high available NetApp Storage, we DO NOT use our Core, for the passing of storage traffic.

This gives us the high availability we need, if the Core is affected, our VMs and SAN are still running.

We use Cisco and HP for our Core and Edge switches to which are VMs are connected, for End User traffic, but ALL storage goes via seperate Storage Network.

There is no requirement to connect iSCSI and SAN to Core.
0
 
LVL 10

Assisted Solution

by:Schuyler Dorsey
Schuyler Dorsey earned 250 total points
Comment Utility
Andrew is completely correct.

However, if you are a smaller business and don't have the capacity to physically separate them, you can still logically separate them.

E.g. Say you have two Core switches stacked. You can have two "storage" switches stacked which connects your NetApp to your servers. How CAN pass DATA through those switches and uplink those switches to your Core without an issue. The key is putting data and storage traffic on separate VLANS.

So if you vlan them out, the switches will NOT pass storage traffic to your core. You wouldn't even have to tag storage traffic on the uplink port and this would prevent traffic from going to the core for sure.

For smaller businesses, this is a perfectly acceptable solution. I actually see this case more often than not in enterprise networks too.

Though I do have a question for Andrew:

If you do not connect your storage switches to the network in any way, how do you manage them? Seems that would limit you to console only unless you put a pc on that vlan, on that switch.
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 

Author Comment

by:wayy2be
Comment Utility
Very interesting.  What I have now is one 3750 as a "Storage Switch". It has vlans for IP storage for my NFS traffic and then vlans for my ESXi hosts. It is also connected to the core so users can access the vm's. I am not sure if I explained this properly before. So I just need to add another switch, yes?  What else can be done to make it better?  Thanks guys
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Add another switch, add and configure Jumbo Frames, switch configuration to multipath.

Storage Switches do not need to be connected to Core, so users can access VMs!

ESXi Servers needed to be connected to core.
0
 

Author Comment

by:wayy2be
Comment Utility
I am not familar with multipath. Where is that configured and how?  Thanks
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
I posted in my first posting above here in my EE Article http:#a39997314
0
 

Author Comment

by:wayy2be
Comment Utility
Sorry guys got pulled off in another direction.  Here is a diagram of what I have.  What do you think?
vmware.jpg
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
looks okay, if just using NFS.

So no iSCSI.

Also enable jumbo frames.
0
 

Author Comment

by:wayy2be
Comment Utility
So the storage switch is my primary concern, should it go down, everything vm goes down with it.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Yes, that's correct, if you have a single switch, all your VMs will go down.

the same as if you SAN goes down, all your VMs will go down.

two single points of failure!

Add another switch, and/or SAN!

Availability/Resilience/Redundancy is all amount, how much money do you throw at a solution!

How much downtime, can the business afford, how long would it take to get another switch.

(you could have one on the shelf, ready configured!)

Also, do the switches, have one PSU or two, aer they connected to different UPS.
0
 
LVL 10

Expert Comment

by:Schuyler Dorsey
Comment Utility
Does your NetApp only have one controller? If it does, I would at least get a second controller for it.

As far as the switch, I would recommend getting a second switch and stacking the two and connecting your environment to both switches. So if one switch fails, everything continues working seamlessly.
0
 

Author Comment

by:wayy2be
Comment Utility
I have two controllers in a HA configuration.  There is only one point of failure, the switch. We have another SAN at our DR site that we snap mirror all of our data to every hour. I am looking at getting 2 - 24 port Brocade 6450 switches, stack them and cross cable them.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Another switch will resolve the single point of failure issue.
0
 

Author Comment

by:wayy2be
Comment Utility
Thanks guys for your replies and time!
0
 

Author Comment

by:wayy2be
Comment Utility
Andrew,

What model Brocade do you normally use?  I have been looking at the 6610 for this environment.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Dell PowerConnect 7024 (re-badged Brocade switches by Dell!)
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now