Best Practice To Utilize Two 10 Gigabit Ethernet Cards For NIC level Availability.

Environment Details:

I have two Dell Rx730 PowerEdge servers on which I have deployed and setup Exchange 2016 server. Both servers are part of one DAG.  

Each server have two NICs of 10 Gigabit Ethernet Cards. Right now only one 10 Gigabit Ethernet Cards on each server is being used. I want to use other NIC as well for NIC high availability.

Questions:

1.  Should we simple plug the cable to another NIC and set the another IP of same network so servers can communicate with each other on two NICs  of same subnet. In this case if one network card fails then server would be reachable through anoter IP or NIC.  Can we do this way? If not then what is reason behined it?
 
2. We can create NIC teaming using these two 10 GB NICs. If NIC teaming is right option as campre to above one then what
     configuration from given below is best practice for Exchange.

1.  Switch Independent configuration / Address Hash distribution
2.  Switch Independent configuration / Dynamic distribution
3.  Switch Dependent configuration / Address Hash distribution
4.  Switch Dependent configuration / Dynamic distribution

Note: We are aware about PA for Exchange ( use only one 10 GB NIC) but as we have two 10 GB NICs on each server so we want to utilize them.
LVL 2
Muhammad AsifSenior Solutions ArchitectAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

BembiCEOCommented:
Hello,

You defined your target as "high availability". A second NIC is often used as a separate / independent network to separate the client traffic from the DAG traffic.  But depended from the load, it can be also a waste of resources or make the things more complicate.

Be aware, that the statement of Microsoft is to "keep it simple", so they define a NIC Teaming as an additional point of failure. Exchange can handle an NIC outage by the DAG.

Just to connect a second cable to the same network infrastructure maybe not a good idea as you need a kind of algorithm, how the traffic is routed through the one or the other NIC and how the traffic comes back.
A teaming configuration gives you more flow control (reliability) and a controlled switch over in case of a single NIC failure.

From the teaming perspective, it depends from your hardware equipment.
Switch Independent configuration means, that the switch is not aware about the teaming, so the NIC team takes control about the traffic over the one or the other NIC. You can use two different / independent switches.
Switch depended mode means, that the switch controls the flow, but you have to use the same switch.

Dynamic distribution has some advantages over Address Hash, especially in switch independent mode.
My recommendation in this scenario would be - if you can use the same switch - to use the switch dependent mode / dynamic distribution.
With two different switches switch independent mode / dynamic distribution.

Some more details about the modes:
https://technet.microsoft.com/en-us/library/mt179276.aspx

But the best recommendation as only as good as the reliability of the underlying hardware. If the drivers and hardware works fine, everything is ok, but the devil is sometimes in the details. So it may be a good idea, to plan some testing in this constellation to see, which impact it produces on the server side as well as on the client side in case of an outage of one NIC.

Also you may observe the currently used bandwidth to get an idea, how many traffic is really produced on the currently used single NIC to get an idea, if you really have an advantage of the second NIC. As I stated before, any additional configuration make the system more complex.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Muhammad AsifSenior Solutions ArchitectAuthor Commented:
Hi Bembi,

Thanks for your reply in detail. Can you please let me know some more details regarding advantages of Dynamic distribution over Address Hash in switch independent mode?

1.  If we go with  switch dependent mode / dynamic distribution then whats it advantages over switch independent mode/ Dynamic distribution except bandwidth aggregation?  We do not need to aggregate bandwidth so now what you suggest switch independent mode/ Dynamic  or switch dependent mode/ Dynamic?

2.  Moreover, in Dynamic Configuration, shall we need to mark both adapters as avtive?
0
BembiCEOCommented:
Hello Muhammad,

Dynamic distribution combines some  techniques and includes Address Hash if needed. And it is more flexible according rebalance. It is the recommended mode for best performance (with two active Nics).

The difference between switch depend and independent is, that the in the first case, the switch handles the inbound traffic to the NICs while in independent mode, the NIC team decides, how to handle the traffic.  If the single NICs are connected to different switches, you have to choose independent mode. If the NICs are connected to the same switch, you have the choice to let the switch or the NIC team decide.
As the NIC team runs on the server, it produces the overhead on the server.

The independent mode has the (rare) disadvantage, that the switch may send packages to an faulty NIC what may produce a short drop out. The package has to follow the route defined from the NIC team and the switch has no chance to reroute the traffic if it detects a faulty NIC.

Both modes are not connected to bandwidth aggregation (both aggregate bandwidth), the question is only, who (switch / NIC team) controls the traffic flow.

Dynamic mode provides better performance with two active NICs. If one NIC is in stand by mode, you loose bandwidth aggregation and the Address Hash mode provides better performance....

From my customers, which used active / passive mode, we made the experience on HP hardware, that the switch over seems to take more time than two active NICs and sometimes produces short outages. But I guess this is a specific experience of this hardware.
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

Muhammad AsifSenior Solutions ArchitectAuthor Commented:
Hi Bembi,

Thank you so much for your assitance and now things are very much clear to us. So we can configure the NIC Teaming as mentioned in attached screenshot for Exchange 2016?

If we configure NIC teaming with parameters LACP along with Dynamic  mode  (as highlighted in attached screenshot)  by plugin the each network adapter cable to separate switches connected with MLAG.   Is it better choice as compare to Switch Independent/Dynamic mode?  

1. I beleive in NIC teaming,  Teaming Mode (Static and LACP) are two different switch dependent  modes, if yes then what mode you suggest  for NIC Teaming Windows server 2016 for Exchange 2016?

2.  How we can proceed with these configurations. I mean we just need to ask the network team to configure two ports (of two network adapters) with LACP and then we need to create the NIC teaming as mentioned in attached screenshot at server OS level?

3: How we can find that LACP is porperly configured at switch level after creating NIC teaming with LACP option at Server OS level?

4.  What is difference between Static and LACP?
0
BembiCEOCommented:
I do  not see a Sceen-Shot ;-)

1.) Static mode need possibly some manual configuration while LACP dynamically identifies the connections between host and switch.  As far as the switch supports LACP. So, if available, you can use it an enable it on the switch ports.

2.) Yes, this would be my imagination.

3.) If LACP is enabled on the switch, the host should (in theory) configure the LACP mode automatically.

4.) See 1.)
Static mode needs some manual configuration steps which should not be needed if LACP is active.
0
Muhammad AsifSenior Solutions ArchitectAuthor Commented:
Hi Bembi,

Thanks for valuable information.  I have now a clear idea on this subject and would proceed according to the discussions here.

Just one last question, when we create a NIC teaming, we see a virtual NIC interface as well. Do we need to configure the NIC protocols as shown in the below screenshot on the virtual NIC interface or on actual NICs?

NIC-properties.PNG
P.S. I have also attached the missing screenshot as a referrence to my above question. I would be using LACP with Dynamic mode as suggested by you.  Have you also use these two modes in any production Exchange environment?

LACP.PNG
0
BembiCEOCommented:
Yes, the virtual network interface is now, what the host machine can see...

For an internal server (inside the domain) it is fine.

IP6 is only needed, if you really use it, means you have a fully implemented IP6 infrastructure.
If you use only IP4, you may try to disable it.
Exchange use IP6 if available.

The physical NICs should not have all the services enabled anymore, rather you configure the settings like IP Addresses and all other options on the virtual NIC for the team.

Yes, we used it in production environment, but as I said before, it sometimes depends on the hardware.
0
Muhammad AsifSenior Solutions ArchitectAuthor Commented:
Hi Bembi,

Thank you so much for all these details and I really apperciate it.  Is there any best practice or method to verify these configurations of NIC Teaming at switch level.

Is there any method of troubleshooting so we can verify that whether the configurations at switch level done by network team correctly or not. In-case of any issue, if they say yes, everything is working fine from our end then how we can double check. I am just asking this question at server administrator point of view.
0
BembiCEOCommented:
Hi Muhammad,

on switch level, you have usually the port statistics. Dependent on the switch type, you have more or less information, high end switches can give you a details view on the traffic, errors and other statistical information. Cheaper switches have at least a port statistic.
Here, if everything is working as expected, you should see that the two teamed ports should have more or less the same amount of RX / TX packages. This is an indicator that load balance is working fine. The error quote on the posts should be more or less zero. A few errors  (mostly retransmissions) are normal, but a higher rate point to a problem.  You may ask your infrastructure team for such a statistic, just to see, what information you can get. They can just make s screen-shot.

On server level, it maybe a bit more difficult, as the server can get only aware about errors, which are recognized by the server. As more as the hardware is involved, as less you can see on the server. LACP you may see during configuration, that teaming setup works with only a few setup steps. You may observe the performance monitors for the NICs so see transmission statistics. If you open performance monitor, you can add  the performance monitor for "Network Interface". There you can see, what traffic is handled by the interface and if you see errors (i.e Package Received Errors or Package Outbound errors should also more or less zero).
Another test on the server side is just, to plug off one of the cables and see, if an Outlook client makes trouble. At the same time, observe the event log on the servers.

The best place to check the physical configuration is the switch statistic. The test on the server is more or less more the practical test, what impact a NIC switch has.
0
Muhammad AsifSenior Solutions ArchitectAuthor Commented:
Bembi has provided  expert level valuable information.
0
Muhammad AsifSenior Solutions ArchitectAuthor Commented:
Thanks you so much Bemi for such a valuable information.
0
Muhammad AsifSenior Solutions ArchitectAuthor Commented:
Hi Bembi,

We have deployed the LACP NIC Teaming according to your recommendations ( LACP/Dynamic mode). However, we are getting the problem to start the Exchange services ( Exchange Frontend Transport, Exchange Transport, Information store) after rebooting the server.  

We have rebooted the server and once server is back online then we have checked the Exchange services and found that Exchange Transport is stuck in stoping as it always shows the status as stoping.

The Microsoft Exchange front-end service is keeping trying to start and sopting it self and never in running status.  Can you please help us in this regard.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.