NetApp SAN switch network design question

I have a NetApp, a switch and two ESXi servers in my rack. The switch in the rack is connected to the core switch for our LAN via ether channel. It looks like this:



[core switch]---------------------------[storage switch]--------------[NetApp]--------[ESXi 1 & ESXi2]  

My question: is this a typical design?  Do most environments have a separate storage switch sitting between their storage and ESXi servers or do they just connect everything to the core of the LAN?

What is the best practice approach?

Thanks
wayy2beAsked:
Who is Participating?
 
Andrew Hancock (VMware vExpert / EE MVE^2)Connect With a Mentor VMware and Virtualization ConsultantCommented:
It could be, but we install seperate storage switches, usually two, in a STACK, and then comnnect at least two paths from every switch to each ESXi server, using Multipath.

(not teamed or ether channel connections, e.g. trunks)

Best Practice is Multipath, and enable jumbo frames make help.

HOW TO: Add an iSCSI Software Adaptor and Create an iSCSI Multipath Network in VMware vSphere Hypervisor ESXi 5.0

HOW TO: Enable Jumbo Frames on a VMware vSphere Hypervisor (ESXi 5.0) host server using the VMware vSphere Client
0
 
Schuyler DorseyCommented:
As Andrew mentioned, you certainly should have two storage switches stacked in this scenario as you have created a single point of failure. If that one storage switch goes down, all of your VMs will go down.
0
 
wayy2beAuthor Commented:
Absolutely I agree and that is what I am planning to do in the next few weeks. Budget was tight last year so we could only afford one switch.  So how does the storage switch stack get connected to the core LAN switch?  What is the best way?
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
Schuyler DorseyCommented:
Typically a port-aggregation. You could do two ports (one from each storage switch) to the core switch.
0
 
wayy2beAuthor Commented:
I did have a vendor tell me to forget the storage switch and connect straight to the core with the ESXi and NetApp. However I did not feel that this was correct. What do you think of that?  The current storage switch is a Cisco 3750. Is that a good switch for this purpose or overkill?

Thanks for the replies guys
0
 
Schuyler DorseyCommented:
Actually.. that switch is BELOW what I normal spec. Cisco 3750's often have difficulty keeping up with storage due to the low cache buffer sizes. It will REALLY affect your switch is your storage environment does a burst.

If we are specc'ing Cisco, we do a MINIMUM of Cisco 3850. If we do HP, we spec HP 3800s.

As far as connecting directly to the core, I see that a lot at small businesses. So I suppose it would depend on the size of your network/business. It is common practice to use a separate switch for storage.
0
 
wayy2beAuthor Commented:
We have around 200 clients and 25 vm's.  What would you suggest in Brocade?  If I bought Brocade would they place nice with our Cisco core, which are also 3750's?
0
 
Schuyler DorseyCommented:
I can't speak on the Brocade side. I have only worked with their fiber switches. Those are okay but I'm not crazy about the mgmt side of them.

It should play nicely as long as you are not using Cisco proprietary protocols such as EIGRP.  In this case, you would have to add another routing protocol to your network such as OSPF.

Any reason you mentioned Brocade? The specific HP and Cisco switches I mentioned would do wonderfully. The HP one would actually be a better performer than the Cisco, has lifetime warranty, lifetime updates and would be much cheaper.
0
 
wayy2beAuthor Commented:
No reason, just looking at all the options. Would the HP play nice with the Cisco core?
0
 
Schuyler DorseyCommented:
Same with the Brocade.. you only have to consider if you are running any proprietary Cisco protocols like EIGRP. HP is an open standards based switch so it will support RIP and OSPF. Cisco DOES allow you to inject EIGRP routes into OSPF if needed.

But I have several environments with both HP and Cisco running without issue.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We use Brocade or Dell Power Connect 7024 Storage switches in a stack, just for storage.

These are not connected to any other networking, it's the storage network is isolated and runs Jumbo Frames.
0
 
wayy2beAuthor Commented:
Andrew:  So how are you connected to your LAN?
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We have a seperate completely dedicated 2 to 4 storage switches, which ONLY NFS and iSCSI traffic passes from ESXi servers to SAN.

These are Storage Switches purchased for the purpose of high available NetApp Storage, we DO NOT use our Core, for the passing of storage traffic.

This gives us the high availability we need, if the Core is affected, our VMs and SAN are still running.

We use Cisco and HP for our Core and Edge switches to which are VMs are connected, for End User traffic, but ALL storage goes via seperate Storage Network.

There is no requirement to connect iSCSI and SAN to Core.
0
 
Schuyler DorseyConnect With a Mentor Commented:
Andrew is completely correct.

However, if you are a smaller business and don't have the capacity to physically separate them, you can still logically separate them.

E.g. Say you have two Core switches stacked. You can have two "storage" switches stacked which connects your NetApp to your servers. How CAN pass DATA through those switches and uplink those switches to your Core without an issue. The key is putting data and storage traffic on separate VLANS.

So if you vlan them out, the switches will NOT pass storage traffic to your core. You wouldn't even have to tag storage traffic on the uplink port and this would prevent traffic from going to the core for sure.

For smaller businesses, this is a perfectly acceptable solution. I actually see this case more often than not in enterprise networks too.

Though I do have a question for Andrew:

If you do not connect your storage switches to the network in any way, how do you manage them? Seems that would limit you to console only unless you put a pc on that vlan, on that switch.
0
 
wayy2beAuthor Commented:
Very interesting.  What I have now is one 3750 as a "Storage Switch". It has vlans for IP storage for my NFS traffic and then vlans for my ESXi hosts. It is also connected to the core so users can access the vm's. I am not sure if I explained this properly before. So I just need to add another switch, yes?  What else can be done to make it better?  Thanks guys
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Add another switch, add and configure Jumbo Frames, switch configuration to multipath.

Storage Switches do not need to be connected to Core, so users can access VMs!

ESXi Servers needed to be connected to core.
0
 
wayy2beAuthor Commented:
I am not familar with multipath. Where is that configured and how?  Thanks
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I posted in my first posting above here in my EE Article http:#a39997314
0
 
wayy2beAuthor Commented:
Sorry guys got pulled off in another direction.  Here is a diagram of what I have.  What do you think?
vmware.jpg
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
looks okay, if just using NFS.

So no iSCSI.

Also enable jumbo frames.
0
 
wayy2beAuthor Commented:
So the storage switch is my primary concern, should it go down, everything vm goes down with it.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes, that's correct, if you have a single switch, all your VMs will go down.

the same as if you SAN goes down, all your VMs will go down.

two single points of failure!

Add another switch, and/or SAN!

Availability/Resilience/Redundancy is all amount, how much money do you throw at a solution!

How much downtime, can the business afford, how long would it take to get another switch.

(you could have one on the shelf, ready configured!)

Also, do the switches, have one PSU or two, aer they connected to different UPS.
0
 
Schuyler DorseyCommented:
Does your NetApp only have one controller? If it does, I would at least get a second controller for it.

As far as the switch, I would recommend getting a second switch and stacking the two and connecting your environment to both switches. So if one switch fails, everything continues working seamlessly.
0
 
wayy2beAuthor Commented:
I have two controllers in a HA configuration.  There is only one point of failure, the switch. We have another SAN at our DR site that we snap mirror all of our data to every hour. I am looking at getting 2 - 24 port Brocade 6450 switches, stack them and cross cable them.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Another switch will resolve the single point of failure issue.
0
 
wayy2beAuthor Commented:
Thanks guys for your replies and time!
0
 
wayy2beAuthor Commented:
Andrew,

What model Brocade do you normally use?  I have been looking at the 6610 for this environment.
0
 
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Dell PowerConnect 7024 (re-badged Brocade switches by Dell!)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.