Basic ESX/vCenter Networking Best Practice

OK...so networking isn't my specialty :) When I implemented VMware in my org almost 2yrs ago, I for the most part just wanted to get it up and going. That being said, all turned out pretty good, except for the networking aspect. For that stuff, I tend to delegate to our Ntwk Engr. I now feel I know enough about VMware/vSphere to get this taken care of.

The issue? I have 4 ESX4 hosts, and 2 hosts in 1 cluster. I have them all configured with only one NIC. Part of this was due to not knowing enough about why to configure other NICs as well as lack of ports on our GB Switch. Now we have room on our physical switch and I want to get my hosts config'd appropriately. So, what I'm looking for is best practice info. I have an idea of what I want to do, but just want to hear some other suggestions from everyone. Currently I have 1 NIC on each host connected for everything. My question, having 2 NICs, how should I configure each host (I'm assuming configuration would be the same for every host, regardless if it's in a cluster or not)? How should I set up failover/redundancy having 2 NICs available? How best should I configure VMotion, etc. Can I do this all through 1 vSwitch or should I use 2? If so, why? I was going to implement NIC Teaming, but want to know how best to set up my ports for VMs, VMotion, and Service Console for failover/performance purposes. Again, I'm on vSphere (v4, but not U1). I don't have Ent Plus so no dvSwitch is possible. Let me know if more info is needed.

Thanks!
LVL 40
coolsport00Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mike ThomasConsultantCommented:
I don't know if there is a best practice but I would do it like this

Say you have a host with 4 nics, configure them independantly then assign them to guest as you see fit, if a guest just needs 1 nic then assign it nic 1 to them, if another guest just needs 1 nic then assign that nic 2 if a guest needs 2 nics assign it 3 and 4 if the next guest needs 2 nics then assign it nice 1 and 2, and so on so you spread it arround a little.



0
coolsport00Author Commented:
That's not exactly what I'm looking for "Mojo", but thanks for the post.

~c
0
kugaconsultantsCommented:
coolsport00, here is what I would do when configuring your ESX4 hosts to obtain redundancy.

To prevent against any host communication failure, you should always have more than 1 NIC dedicated per subnet per host.  Each host should have at the minimum the following.

2 NICs for LAN
2 NICs for iSCSI
2 NICs for vMotion

Since you have two NICs for each subnet, your next objective is to make sure each is connected to a separate physical switch.  This will now ensure you have switch redundancy and not just link redundancy.  To make things easiest, have a pair of switches for LAN, another for iSCSI, and another for vMotion (or vMotion can be shared with LAN switch but on a separate VLAN).  Here is how it should be connected.

LAN Switch (uplink with via stacking modules or multiple gigabit uplinks in LAG)
- ESX Host 1 LAN NIC 1 connects to LAN Switch 1
- ESX Host 1 LAN NIC 1 connects to LAN Switch 2
- ESX Host 2 LAN NIC 1 connects to LAN Switch 1
- ESX Host 2 LAN NIC 2 connects to LAN Switch 2

iSCSI Switch (some storage vendors do not recommend uplinking iSCSI switches because it will create duplicate storage paths)
- ESX Host 1 iSCSI NIC 1 connects to iSCSI Switch 1
- ESX Host 1 iSCSI NIC 1 connects to iSCSI Switch 2
- ESX Host 2 iSCSI NIC 1 connects to iSCSI Switch 1
- ESX Host 2 iSCSI NIC 2 connects to iSCSI Switch 2

vMotion Switch / LAN Switch with VLANs
- ESX Host 1 vMotion NIC 1 connects to vMotion/LAN Switch 1
- ESX Host 1 vMotion NIC 1 connects to vMotion/LAN Switch 2
- ESX Host 2 vMotion NIC 1 connects to vMotion/LAN Switch 1
- ESX Host 2 vMotion NIC 2 connects to vMotion/LAN Switch 2

If you cannot have dedicated switches but have a VLAN capable stackable switches (e.g.  Dell PowerConnect 6248 w/stacking modules), create 3 VLANs and label them as LAN, iSCSI, and vMotion.  You always want to separate your LAN, iSCSI, and vMotion networks for optimal efficiency.  Connect your ESX4 hosts as stated above and treat each VLAN as a different switch.  Be sure to spread your links across the switches in the event of a switch failure.

Once you have your physical network configured, the next step is to configure each ESX4 host.  Each network will require it's own vSwitch to virtually segregate each network.  Each vSwitch requires a port group.  The first called LAN, second called iSCSI, and third called vMotion.

For each vSwitch, create the following Service Consoles and VM Kernels.

LAN vSwitch
- 1 x Port Group for LAN
- 1 x Service Console

iSCSI vSwitch
- 1 x Port Group for iSCSI
- 1 x Service Console
- 2 x iSCSI Kernel (one per iSCSI NIC)

vMotion vSwitch
- 1 x Port Group for vMotion with VMotion enabled

After each ESX4 host is configured, make sure to configure NIC teaming on LAN ports and configure multipathing for iSCSI ports for each host.

I hope this gives your a good start on configuring a fault tolerant solution.
0
Powerful Yet Easy-to-Use Network Monitoring

Identify excessive bandwidth utilization or unexpected application traffic with SolarWinds Bandwidth Analyzer Pack.

coolsport00Author Commented:
Thank you "kugaconsultants", but that is WAYYY more than what I'm asking. I applaud the effort! Let me be just a bit more specific. I simply just want to know how to configure my ESX/vCenter networking (vSwitch(es), NICs, Port Groups). The physical network will not be modified in any way. Also, I have a FC SAN, not iSCSI (forgot to mention that above). So, what I'm wanting clarification on is how best to configure vCenter/ESX networking with only 2 NICs per host. Again, I have 4 hosts, but 2 of those will be (are) in a cluster, but I don't think that really matters. Keep in mind I did read through the Netwkg section of the ESX Config Guide...so I think I have an *idea* what to do, just want any 'best practice' suggestions, if there are any.

Thanks!
~c
0
jakethecatukCommented:
her dude,
If you've got two NIC's per server and that is all you have, then the simplest way would be as follows: -

NIC1 - Service Console, VMotion, VMKernel
NIC2 - virtual machines

NIC1 will not see much traffic and could be seen as a waste but you are putting a clear demarcation between the management network and production network.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
bbnp2006Commented:
I would configure my networking the following way with only 2 pNIC on my side:

Team both NIC first then:

vSwitch 0 --> service console port and VM network: use pNIC1 as Active, and pNIC2 as Standby
vSwitch 1 --> VMKernel Port for vMotion: use pNIC2 as Active and pNIC1 as Standby

This way you have failover capacity that either of your NIC fails, the traffic will fail over to the other NIC. Of course you have to configure VLAN on your physical switch side to trunk the 2 physical ports where your teamed NICs are connecting to.

Food for thoughts mate.
0
jakethecatukCommented:
-bbnp2006...when you say 'Team both NIC first', think you'd better explain that one in more detail as I'm guessing you mean at a physical switch level as this can't be done on ESX.
0
bbnp2006Commented:
sorry guys, please ignore that "Team both NIC first" line :)
0
jakethecatukCommented:
and I think we need to ignore 'and pNIC2 as standby' and 'and pNIC1 as standy' as well as you can only allocate one NIC to one vSwitch.

looks like after the edit, bbnp2006 came up with the same suggestion as me :) LOL...

only teasing bbnp2006
0
bbnp2006Commented:
LOL... that's true jake :)
coolsport00, have you also considered using just ONE single vSwitch and put all 3 port groups in one single vSwitch, then add both pNICs on this single vSwitch, then you can override the failover configuration on the NIC level for each port group.
Food for thought.
0
bbnp2006Commented:
and to backup my design using articles from VMWare blog:
http://blogs.vmware.com/networking/2009/04/index.html
Quote:
"It is a common misconception that increasing the number of virtual switches used in an ESX system allows for greater performance through more physical CPU's being utilized in parallel for driving the I/O. For example, instead of having a single virtual switch with two physical NIC's connected to it, some customers choose to create a separate virtual switch for each physical NIC, in hopes of getting performance benefits."

"The bottom line here, of course is that the number of virtual switches does not affect network performance. In most cases one vSwitch with proper use of VLANS (VST mode) and port group override of NIC teaming policies is quite ample."

Thoughts?
0
bbnp2006Commented:
so here's my thought:

vSwitch0 --> add pNIC1 and pNIC2
Then add 3 port groups to vSwitch0:
SC: Active: pNIC1, Standby: pNIC2
VM Network: Active: pNIC2, Standby: pNIC1
VMKernel(VMotion enabled): Active: pNIC1, standby: pNIC2

0
coolsport00Author Commented:
What I have now is 1 NIC with everything. I'm such a small org that it, believe it or not, runs pretty fine (performance-wise). But obviously there is no failover. So, NIC Teaming on the vSwitch is what I was thinking about doing, then for the most part, configure my ports as what you guys suggested. So, my plan was to use (still) just 1 vSwitch. If there is a reason why I should use 2, I'm open as to why. Oh, and I want to configure failover for the service console....how could I do that, too?

Thanks guys!
0
jakethecatukCommented:
the only reason I can think for not using one vswtich vs. two would be bandwidth utilisation.

picture the scenario, you have one vswitch with both nics.

you kick off a vmotion and when you do that, it could use the same NIC as all your production machines and impact on your network.  having seperate vswitchs would prevent that.

but to counter that, having two vswitches gives you no resiliance if you lose a NIC.

Maybe the way forward would be one vSwitch with VLAN's to split the traffic out.
0
coolsport00Author Commented:
Actually, I think the only reason why I would create a 2nd vSwitch is for testing. I wouldn't allocate a pNIC to it...just VMs (so yes, I would keep the VMs isolated). I would maybe create a test domain or something on this 2nd vSwitch. And, as I figured, I didn't think there was performance benefits to having a 2nd vSwitch. See guys....easy points...ha!
0
bbnp2006Commented:
All you need to do now coolsport00 is to add the 2nd NIC to your existing vSwitch, click on each port group that is already configured on your vSwitch, click on Edit, go to the NIC Teaming tab (last tab), then check the box saying override vSwitch failover order. You can simply use pNIC1 for all your port groups just like how it is now, then MOVE pNIC2 down to the "Standby" section, so when your first NIC fails,  all your traffic automatically fails over to the 2nd NIC. Is that kind of NIC failover you are looking for mate?

0
coolsport00Author Commented:
Here's a screenshot of what I have...
ESXNtwkConfig.bmp
0
bbnp2006Commented:
Mate, i have couple of previous customers running the exact same setup as you have right now :) Doesn't hurt at all to add the other NIC to the Standby mode just in case your vmnic0 fails.
0
coolsport00Author Commented:
I thought it would be as simple as that...but just wanted to check. As I stated in my orig post, networking isn't my 'thing'. I think I make it harder than it is, or it's simply that I don't deal with it much..my Ntwk Eng does. :P hahaha

Thanks fellas!
~c
0
bbnp2006Commented:
again for the benefit of bandwidth, you can always seperate the traffic of VM Network and your vMotion/SC traffic using VLAN and override the failover order, so when both NICs are functioning, your network traffic are seperated.
0
coolsport00Author Commented:
How would that look?
0
bbnp2006Commented:
Gd luck coolsport00! Does your Ntwk eng say jokes like "There's nowhere like 127.0.0.1?" LOL...
0
coolsport00Author Commented:
And "jake...", using your #28453477 post, which is what I was thinking, I can still configure NIC Teaming on the vSwitch correct?
0
coolsport00Author Commented:
I'm always asking for pizza and, for some reason, I never get it! :P
0
coolsport00Author Commented:
And, I'll have to remember that one "bbnp..." HAHAHAHAHA
0
coolsport00Author Commented:
Oops...one last question before I close this and give out the pts...does making the networking changes require ESX host reboots?
Thx!
0
jakethecatukCommented:
No - not normally anyway.
0
coolsport00Author Commented:
And btw "bbnp...", to answer you from post #28455774, yes...that's what I was looking for? :)

Great...thanks!
0
coolsport00Author Commented:
So "jake...", were you saying I couldn't use NIC Teaming earlier? (see post # 28453890) Teaming can be done in ESX...unless there was something else you were meaning?
0
jakethecatukCommented:
Teaming can be done - but only when the NIC's are allocated to the same vStwitch.  bbnp2006 made a slight error on his post above #28453746 - I was merely correcting the error.

It's a pity that under ESX you can't agregate the available bandwidth when you have more than one NIC on vSwitch (i.e. have 2GB bandwidth when you have two NIC's).

0
coolsport00Author Commented:
Ahh...yes; he mentioned having 2 vSwitches, of which I'm not going to have...I'm just gonna use 1. Thanks. Yeah...that would be nice...maybe someday :)

~coolsport00
0
coolsport00Author Commented:
Good stuff; thanks guys!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.