vmware esx ent nics and portgroup question

Posted on 2009-04-05
Medium Priority
Last Modified: 2012-05-06
Hi Guys,
i got 6 nics on each host esx 3.5 ent server
we got 2 hosts
i want to know what portgroups needs configuring and how?
i want on the esx side of the configuration with different VLAN configuration example please.
can someone give me a complete sample setup please
i want to know, where i can put the Hosts, i mean
SC---which VLAN ----Which vnic?---Ip addredd?
Vmkernal(iSCSI)-----which VLAN ----Which vnic?--ip add
Vmkernal(VMotion)--------which VLAN ----Which vnic?---ip add?
SAN----Ip add?
VM's-----which VLAN ----Which vnic?---ip add

please advise

Question by:APPIREDDY
  • 12
  • 5
  • 4
  • +1
LVL 32

Assisted Solution

nappy_d earned 200 total points
ID: 24071664
NIC1 - Service Console, VLAN10
NIC2 - Service Console VLAN10
NIC3 - VMKernel, VMotion VLAN11
NIC4 - VMKernel iSCSI, VLAN12
NIC5 - Production , must connected to the VLAN that your client workstations are located on.
SAN - VLAN12 for iSCSI communication to your ESX servers

  • You can decide for yourself, what IP address schemes you want to use.
  • You should get additional NICs for your guest operating systems to share.  

Assisted Solution

qualchoice-it earned 200 total points
ID: 24071738
Here's a run down on how we have it setup at my company

VSwitch0-vmnic's 1, 2, 0, and 4 are dedicated to my VM's on VLAN 7 and Service Console

VSwitch2-Vmnic 3, is hosting my Iscsi. VLAN 10

As far as vmotion, just go in and enable HA and DRS, that will create a cluster and allow you to cold migrate or hot migrate VM's from one host to another.

Author Comment

ID: 24072177
Hi Qualchoice-it,
Thank you very much.i'm really for your answer.just hetting there. can u clarify me few things.is it enough to crea one vmkernal iscsi and enable the vmotion on the same VLAN as iSCSI one or should i have to create the vmkernal+Vmotion as separate to vmkernal+iSCSI?if i want to do the following how it goes?

vsw0---vmkernal+vmotion-------VLAN 1- no routing to any VLAN's
vsw1---vmkernal+iSCSI-------VLAN 2
vsw2----VM's-----VLAN 3-----connected to existing LAN Network
do i need to create a separate VLAN  called VLAN 4 for Service console?, if so should i enable the routing  between  this VLAN 4 and VLAN 2? please clarify me.
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.


Author Comment

ID: 24072239
Hi Nappy_d,
sorry i missed your reply. thank you so much for the reply. the solution you gave looks almost close.but i have some more doubts please.i need this configuring tomorrow.also i don't under stand why we need two console port groups in the same VLAN?
please let me know if i'm wrong here with the following

vsw0---vmkernal+vmotion-------VLAN 1- no routing to any VLAN's---vmnic4
vsw1---vmkernal+iSCSI-------VLAN 2----vmnic 2 and vmnic 3
vsw2----VM's-----VLAN 3-----connected to existing LAN Network---vmnic0 and vmnic1
vsw3----Service Console----VLAN 4--- vmnic 5
My question here is
should i enable VLAN routing beween VLAN 4 and VLAN 2? is it necessary? what about the above configuration? please let me know asap. i really thank both of you.

Expert Comment

ID: 24072241
Our vmotion is going across our Iscsi subnet VLAN 10, we also have a Seperate Service Cosole for Iscsi all apart of the same Port group on Vswitch 2

No your Service console can exist on the same VLAN as your VM's no need for a serperate VLAN there

See screen shot


Author Comment

ID: 24072307
Hi Qualchoice-it,
thank for that.as best practice and security reasons, people advise us to keep the SC on a completely different VLAN to Prodution VLAN(where VM's will be placed).I'm happy to kep the SC on to a different VLAN like i said in the above example VLAN-1. but i want to know, does it needs access to any of the other VLAN's?nappy_d has given me some clues, but not clear on the VLAN routing and communication requirements. also which VLAN's i need to isolate completely.waiting for some help on this now.thanks for the evry imput guys.

Expert Comment

ID: 24072314
i would keep your service console on the same VLAN as your VM's, also you do realize that your only dedicating two NICS for your VM's? I dont know how many VM's your going to have but that seems kind of skinny to me, but you can always add more later.

Expert Comment

ID: 24072388
I've seen SC seperated to its own Vswitch but not its own V-Lan, but if thats what your doing then I would say yes, you need to make sure whatever VLAN you put your Service Console on, that it can See your VM's as well

Author Comment

ID: 24072863
keeping SC in VM's VLAN means, giving access to any machine in production network to connect to ESX server.what advantage we will get by keeping the SC to a separate vswitch?if the SC is in Production VLAn means we are keeping ESX in the Production VLAN.do you think it is absolutely fine?we also have to kee the Virtual Center Server in the same VM's VLAN.Am i correct?

Author Comment

ID: 24078794
i managed to configure as attached. can someone have a look at the config let me know if anything is wrong?one thing i don't understand is when i try creating the second service console port in VLAN 7, (same VLAN as iSCSI), the default gateway for this one is showing the first SC port which is in VLAN 30.is it OK are should i have to do anything? please advise

Expert Comment

ID: 24078883
only one SC permitted per vswitch.  Whats vSwitch1's purpose?

Expert Comment

ID: 24078937
If you don't have security considerations to weigh into the decision it's all quite simple.
Dedicate 2 NICs to your Service Console, these 2 NICs will quite happily deal with Service console traffic and Vmotion traffic.eg. VLAN10 192.168.10.x (you'll need 2 IP's) so for Service Console and for vMotion.
Dedicate 2 NICs to your storage network. eg. VLAN11 192.168.11.x ip of
Dedicate the final 2 NICs to your Guest session traffic. eg. VLAN12 192.168.12.x
nappy_d has suggested you put 2 ports into each vSwitch so you have redundancy in ports. This is quite necessary..
I would also suggest you separate you port grouping so no 1 NIC is supplying both ports to a VLAN.
NIC card 1 has vmnic0 and 1 on it
NIC card 2 has vmnic2 and 3 on it
NIC card 3 has vmnic4 and 5 on it
therefore I would use
vmnic0 and 3 for the service console and vmotion.
vmnic2 and 5 for the IP Storage Traffic.
vmnic4 and 1 for the vm guest traffic.

Hope this helps.

Author Comment

ID: 24079144
hI gUYS,
THANKS FOR THE REPLY. VSWITCH1 is doing vmotion.
Actually i gor 2 PCI NIC cards
1 has got 2 ports
and another one has got 4 ports. total of 6 physical ports.markzz i didn't understand the NIC1 e.tc at the bottom.
my goal is isolating/separating the iSCSI traffic to it's own VLAN and i think i need to configure the vmotio on to a different VLAN (if i'm right).
how can i achive both isolation of iSCSI traffic and VM's traffic, also if possible vmotion traffic.
can some one clarify me about creating second service console port?when i try creating this i could only see the first SC's default gateway.i'm really sorry. i can not get this working.i'm desparely looking to finish this asap. i'm tired of this

Author Comment

ID: 24079344
can somebody help me please.............
LVL 32

Expert Comment

ID: 24079729
Here is what mine looks like.

  • Service Console -Two NICs for redundancy
  • My Production VSwitch - Two NICs for redundancy
  • I have a testing Network
  • I have a DMZ which is not in production yet
  • I use a Fibre channel SAN but sinc you are using iSCSI I would put one of the NICs into a VLAN with the iSCSI devices for segmented Traffic
  • I would then put another NIC into a separate vlan for vmotion
  • Really and truely though, I would thin you need 8 NICs for full redundancy; that's 2 NICs for SC, 2 NICs for your Production, 2 NICs for your iSCSI and 2 NICs for your VMotion.
  • I don't have that right now until we upgrade our servers later this year.

Author Comment

ID: 24086832
THAKS FOR THE REPLY.i did as followes and it worked OK.does this looks OK in terms of traffic segrigation?
vswitch0--------SC----vmnic0----VLAN 30
vswitch1--------iSCSI-----vmnic1,2-----VLAN 7
vswitch2-------VM--------vmnic3,4-------VLAN 30
vswitch3-------Vmotion,SC------vmnic5------VLAN 10, VLAN 7
SAN--------VLAN 7
This works fine for me. my question is do i need second SC on VLAN 7 on vswitch3 inorder to see iSCSI to work?or there is no need of second SC on the same subnet as iSCSI? this is biggest question and i'm confused with.
LVL 32

Expert Comment

ID: 24087509
Hmm, looks fine BUT I don't know if I would make vswitch3 do both your SC and vmotion.  If you have the spare PCI slot, get another 4 port NIC and then add each NIC to the various vswitches for redundancy.

You already have vswitch0 for SC leave it as such, don't put vswitch3 to support both sc and vmotion.

Expert Comment

ID: 24090389
Although it may not be ideal, what happens if you drop connectivity to the SC for a few seconds?..Nothing.
The point is you have 2x Gb nics, in a team, which is far more bandwidth than ESX performing multiple vmotions and SC management task can utilise. Even storage vMotions aren't an issue, It seems we get about 1 to 1.5GB per minute when cloning or performing cold migrations, which equates to about 130Mb (bit not byte) per second, as you see this is less than 20% utilisation on a single Gb port.
I can only assume there are other limiting functions occuring but the NIC bandwidth doesn't appear to be one of them.
Actually cold clones are via the Storage Medium in my case 2x 4Gb FC, as are storage vMotions. The SC  plays a management roll.
We have about 30 ESX host configured SC and vMotion in the same way.
I know ESX manages guest traffic as session based, I suspect SC and vMotion traffic is session based as well,.again lending evidence as to why it works well.
I couldn't say I've proven this with the SC/vMotion trafffic, there has been no need to date.
The main objective here is to ensure you have NIC port redundancy which means every vSwitch should have 2 NIC interfaces assigned to it.
OK so you have 1 quad interface NIC and 1 Duel Interface NIC.
This makes the breakup a little more difficult.
I was assuming you had 2 onboard interfaces and 2 addin cards with 2 interfaces on each.
Given the new info.
I would likely use this config:-
Quad NIC Port A "Service Console/vMotion" vSwich0
Quad NIC Port B "Service Console/vMotion" vSwich0
Quad NIC Port C "IP Storage"  vSwich1
Quad NIC Port D "VM Guest" vSwitch2
Duel NIC Port A "IP Storage"  vSwich1
Duel NIC Port B "VM Guest" vSwitch2
This at least configuration gives you NIC and Port redundancy for the most critical vSwitches being Storage and Guest traffic.
As mentioned Service Console traffic and connectivity is very important but if it drops a packet or 2 ESX won't fall over.
If IP Storage drops a few packets things get very untidy. Guest traffic isn't such an issue but you'll have complaining users which can be worse.

Accepted Solution

markzz earned 1600 total points
ID: 24090543
Just reading a little more.
Are you using a routing module in your switch for routing between vlans or are you intending on using ESX??
Routers perform routing much more effiently than ESX.
If you don't have a router or switches with routing modules you could look at vyatta, but you'll need more NICs.
Just to recap on the vlaning
Quad NIC Port A "Service Console/vMotion" vSwich0 vlan10 192.168.10.x
Quad NIC Port B "Service Console/vMotion" vSwich0 vlan10 192.168.10.x
Quad NIC Port C "IP Storage"  vSwich1 vlan11 192.168.11.x
Quad NIC Port D "VM Guest" vSwitch2 vlan12 192.168.12.x
Duel NIC Port A "IP Storage"  vSwich1 vlan11 192.168.11.x
Duel NIC Port B "VM Guest" vSwitch2 vlan12 192.168.12.x

Id suggest keeping you vlans separate, use a router or vyatta.
This is not for security reasons (as you have not stated there are any security requirements) this is simply to isolate your traffic.
I would suggestvlan10 will need to be able to see vlan12 and vlan12 neeeds to see vlan10 but vlan11 doesn't need a gateway at all.

Author Comment

ID: 24095554
this looks excellent. i'll configure accordingly and post back my config mate. you are genius.thanks a ton everybody who helped. i'll post back the config in few hours time.I have Cisco Layer 3 switch for VLAN routing, so no need of ESX acting as router.
many thanks.

Author Comment

ID: 24105553
FInally it looks like i have finished, but i want to post my config here to double check.also i want to install the second esx server and hopefully i need to do the same.but i have one doubt, i know i need to create a trunk and do a port channel on cisco switch, which i did configure for the first ESX server. i have created two trunks with portchannel. what i want to know is, suppose, if Gig 0/1 and Gig 0/2 are in port-channel group 1 for the Firsthost(vmnic 2, vmnic 4), can i put Gig 0/3 and Gig0/4 into the same port-channel group for similar traffic say iSCSI , those will be connecting to vmnic 2, vmnic 4 on the second host? or i have to create another port channel and put the Gig 0/3 and Gig 0/4 into that group?
My Configurations on first Host are as folllowes:
Quad Core NIC 1-----SC/Vmotion----vSW0-----vmnic0----VLAN 30-----
Quad Core NIC 2-----SC/Vmotion----vSW0-----vmnic5----VLAN 30-----
Quad Core NIC 3-----iSCSI----vSW1-----vmnic4----VLAN 7-----
Quad Core NIC 4-----VM Gust----vSW2-----vmnic3----VLAN 6-----192.168.3.X

Dual Core NIC 1--VM Gust----vSW2-----vmnic1----VLAN 6-----192.168.3.X
Dual Core NIC 2--iSCSI----vSW1-----vmnic2----VLAN 7-----192.168.4.X
There is dedicated Switch for iSCSI traffic and no routing to any other Vlans
there is a inter Vlan routing enabled between VLAN 6 and VLAN 30.

Many thanks for your valuable help.also how i can i see, my CD-rom drive on the
ESX host mapped to which drive?because i need to install a VM.also i can make an iso  from windows 2003 CD?
thanks a lot


Author Comment

ID: 24108327
now can not see SAN targets from esx SERVERS.i have tried restrting ESX and Openfiler SAN server, but no use.can someone help me....please

Expert Comment

ID: 24108531
The config looks good.
You will need to create another chanel group for the 2nd host.
Think of it like this.
If the ports are in the same chanel group when host 1 sends data to the switch the switch which sends to the SAN...... the switch will reply via the chanel group, therefore host 1 has a 50% chance of recieving the data and host 2 has a 50% chance of recieving the data.
If I remember correctly you will also need to set NIC teaming "based on router hash" (you will see it under NIC's on the vSwitch.

Author Comment

ID: 24183835
thank you very much for all your support and help. i manged to configure everything now.one thing it was missing in my above configuration was  i didn't configure SC inaddition to ISCSI on VSW1, hence failing to contact the SAN from the host.i think most of the people miss this.But that is the only bit i did with Ip address in the same subnet as iSCSI, which works brilliantly.i going to share the points here.I once again appreciate everyone who really helped me lot.
thanks guys

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

When rebooting a vCenters 6.0 and try to connect using vSphere Client we get this issue "Invalid URL: The hostname could not parsed." When we get this error we need to do some changes in the vCenter advanced settings to fix the issue.
If you need to complete a Physical to Virtual (P2V), Virtual to Virtual (V2V) conversion to a VMware product (VMware Workstation, Player or VMware vSphere (ESXi) ) for FREE, then there is some good news...
Teach the user how to install ESXi 5.5 and configure the management network System Requirements: ESXi Installation:  Management Network Configuration: Management Network Testing:
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:

589 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question