Solved

Vmware Physical Network and Storage Connectivity

Posted on 2014-03-29
24
590 Views
Last Modified: 2014-03-31
in most of Vmware environments when they bring an ESX host to the network, they use Network and storage redundancy.
I mean they connect 2 fiber channel adapters from ESX host to 2 separate fiber Channel Network Switches
and they connect 2 finder channel adapters from ESX host to 2 separate fiber channel Storage Switches.

I would like to know what is the best practice for Network and Storage physical connection while  taking into account the need to separate Management Network (Which I guess is the traffic that goes between Vcenter and ESX hosts), vMotion, sVmotion, and Management traffic ( I guess it is for VMHA) and VM Network I guess (traffic between Virtual Machines).

Thank you
0
Comment
Question by:jskfan
  • 12
  • 4
  • 3
  • +2
24 Comments
 
LVL 24

Expert Comment

by:Mohammed Khawaja
Comment Utility
What you need for networking is plenty of network cards.  As an example with 4 NICs, I would make NIC 1 for management and backup for vmotion and backup.  NIC 2 would be the reverse of 1.  I would team NICs 3 and 4 and I have redundancy.  If I had 8 NICs then I could have dedicated NICs for management, vmotion, backups and vm traffic.   As always have multiple storage controllers and with fiber channel cards, most SANs support active paths.  If you are using iSCSI then you could have multiple NICs as well as teamed NICs.
0
 

Author Comment

by:jskfan
Comment Utility
if I understand VM Network (Virtual machine port Group) is the one that has the heavy traffic. Correct ?
if so then I believe we need to dedicate VM Network  to the 2 Fiber channel physical Nics

for vmotion we can add ethernet NIC
for Management Network ( Vcenter to ESX communication) we can add ethernet adapter
for Management Traffic( VMHA) we can add ethernet adapter
0
 
LVL 13

Assisted Solution

by:Abhilash
Abhilash earned 125 total points
Comment Utility
You will not need two cards for management. One NIC is enough. You will need one for vMotion and One or Two for VM traffic depending on the kind of traffic you generate and network redundancy you choose.
We run out environment with 4NIC and 2HBA's on each server. That gives us redundancy for both storage and Network.
Of course we also have redundant switches and also redundancy is take care of at port level on the storage array.
0
 
LVL 61

Assisted Solution

by:gheist
gheist earned 125 total points
Comment Utility
One of most important considerations in vmware network design is complete isolation of vmware service network and user networks.
You sail you want redundancy - so 2 adapters for user networks and 2 for system
In system one is primary for vmotion another primary for storage, and 1 control network on each.

Since you have separate storage redundancy setup you can allocate 1 adapter for system uses (as it loses control connection it keeps wortking, or vmotion is cancelled.)
0
 

Author Comment

by:jskfan
Comment Utility
I guess we are not in the same page regarding Terminology, When I say:

**VM Network = I mean Virtual Machine port Groups, to my understanding this handles the communication between Virtual Machines through ESX hosts and Vcenter with ESX hosts.
I believe the 2 fiber channels that connect ESX host to the Network Switch will be enough to do the job.

***Vmotion = I mean the moving of the Virtual machines between ESX Hosts (the body of the VM stays in the same datastore, just RAM/CPU/NIC traffic will switch to another ESX host). I thought this will be handled by the 2 Fiber Channels that connect ESX Host to Fiber Channel Storage Switch.

**Management Traffic= I mean VMHA, to my understanding this handles the heartbeat between Master ESX host and other ESX Hosts as well as VM machines.
We can either add an Ethernet adapter to dedicate VMHA traffic to , or just make VMHA traffic goes through Network Adapters already assigned for vMotion or VM Network.


I could be wrong in the Terminology and Role of each component, if so,  please correct.

Thanks
0
 

Author Comment

by:jskfan
Comment Utility
please, just to make it clear, when you mention any port group, Management Network, Management Traffic,etc….., put the explanation between parenthesis.
Vmware is confusing with its terminology.

Thanks
0
 
LVL 13

Expert Comment

by:Abhilash
Comment Utility
FC Adapters are for storage traffic. Not network traffic. You will need NIC cards for network/Ethernet traffic.
Your terminology is correct. Management traffic will also involve the traffic that is used to access the host from its IP.
Your terminology is correct but why are you bringing fiber channel adapters. If you replace them with NIC cards in your comment you are absolutely right.

Or if i have to sum it up for you. You will need a NIC  each for Management traffic and vMotion.
You will need another pair for VM traffic.
Totally 4 of them for management, vMotion, and VM traffic.

Hope i made it clear for you.
0
 

Author Comment

by:jskfan
Comment Utility
We have 4 HBAs , 2 goes to 2 separate FC Network Switches and 2 goes to 2 separate FC Storage Switches.
0
 
LVL 13

Expert Comment

by:Abhilash
Comment Utility
Can you confirm if they are FCoE NICs?
You will not have just HBA's.
0
 
LVL 24

Assisted Solution

by:Mohammed Khawaja
Mohammed Khawaja earned 125 total points
Comment Utility
VM network is used for networking and not storage.  Your FC cards are HBAs and used for storage which has nothing to do with VM network and is used by your Data Store.  Do not confuse networking with storage.  For VM network you need Ethernet adapters.  For iSCSI storage, you use Ethernet cards but it is not used for your VM Network usage.
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
I wonder what is fiber channel network switch... Is it 10GbE ethernet or something less brainy?
0
 

Author Comment

by:jskfan
Comment Utility
I am not at work…

But I am sure that the ESX host has 2 FC cables going to 2 separate Network switches and 2 other FC cables going to 2 separate switches for Storage.

The FC cables for Network and Storage look different though
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 13

Expert Comment

by:Abhilash
Comment Utility
Yes they should look different. One pair would be either 1Gig or 10Gig Ethernet cables :)
And other pair would be FC uplinks.
0
 

Author Comment

by:jskfan
Comment Utility
fcfc
0
 

Author Comment

by:jskfan
Comment Utility
The black cables for Storage I believe and the orange ones for Network

but they are plugged to 4 HBAs….2 HBAs go to 2 separate Switches for Network
2 other HBAs go to 2 separate switches for storage
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
Exactly other way around. Black one has SFP module on both ends.

So you have 10GbE
Set it in one of failover modes and use netioc to make sure two vmkernel services and user network dont clash.
0
 

Author Comment

by:jskfan
Comment Utility
what do you mean by vmkernel services and user network…..these are other terms of technology….(:)
0
 
LVL 24

Expert Comment

by:Mohammed Khawaja
Comment Utility
Both pictures are fiber cables used for connecting to storage.
0
 

Author Comment

by:jskfan
Comment Utility
In Fact these are the standard traffics that exist in Vmware:

**Management Traffic: Host to Host communication including VMHA and hosts to Vcenter communication.
Though in some Terminology Management traffic is for VMHA the other traffic is Management Network (Host to host to Vcenter)

Well do I need one Vmkernel port to Management Network and one Vmkernel to Management traffic ?

**Vmotion Traffic
do I need separate Vmkernel for Vmotion?

**IP Storage Traffic
Do I need separate Vmkernel for Storage Traffic?

These are the most common traffic (I did not cite Fault tolerance Traffic)

How many NICs should I have for the traffic separation and redundancy, considering that I already have 4 HBAs in each ESX host, 2 connected to 2 separate Network Switches and 2 to 2 Storage Switches
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 125 total points
Comment Utility
We double up ALL network services for ethernet and fibre channel.

so we have at least 2 nics per Management Network, vMotion, iSCSI, Virtual Machine Network, these could be actual physical network connections, or two nics, and many VLANs, if using 10GBe.

if using 1GBe, it's many nics for resilience, with no VLANs.

e.g.

nics!
You will need a pair for vMotion (VMKernel) and a pair for iSCSI Storage Traffic, but do you use iSCSI ?

if you are using FC, you do not need a pair for iSCSI Storage Traffic, because your storage traffic goes via FC.

But vMotion goes through ethernet.
0
 

Author Comment

by:jskfan
Comment Utility
it looks Good…I do not see where VMHA goes
0
 

Author Comment

by:jskfan
Comment Utility
I guess VMHA can go with VM Network
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
Management Network!

HA Traffic (e.g. communications and heart beat)
0
 

Author Closing Comment

by:jskfan
Comment Utility
Thank you Guys
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
Teach the user how to rename, unmount, delete and upgrade VMFS datastores. Open vSphere Web Client: Rename VMFS and NFS datastores: Upgrade VMFS-3 volume to VMFS-5: Unmount VMFS datastore: Delete a VMFS datastore:
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now