• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1563
  • Last Modified:

Why Is My NLB cluster Only Accepting ICMP traffic?

Points of My Scenario:
1. I am admin for a 2003-based domain.
2. Two member Windows 2008 R2 servers are in a network-load-balanced cluster
3. The NLB cluster's purpose is to support a website
4. The port rules allow all tcp port (0 - 65535): see attached screenshot (NLB_Sttgs.PNG)
5. This nlb cluster has stopped accepting web traffic (or any other) - only ICMP is accepted (I can ping the cluster name and IP)
Question:
How do I resolve to get web traffic through the nlb cluster again?
NLB-Sttgs.PNG
0
waltforbes
Asked:
waltforbes
  • 12
  • 9
1 Solution
 
page1985Commented:
Are either of the hosts listed as "Stopped" or "Suspended"?

When you open the Windows NLB manager, do the hosts in the cluster show up as green, yellow, red, or grey?

Is port 80 open on the Windows firewall?

Is the NLB cluster mode "Unicast", "Multicast", or IGMP?

Is the switch configured to allow promiscuous mode for the ports where you connect these servers to?  Is it virtual or physical?
0
 
waltforbesSenior IT SpecialistAuthor Commented:
Hi Page1985. Answers to your questions:
1. No host is stopped or suspended
2. In NLB manager, the hosts show up as green
3. Windows firewall is completely disabled (for all network profiles)
4. NLB cluster mode is "Unicast"
5. See ADDITIONAL INFO section
--
ADDITIONAL INFO
1. The servers are virtual...
----->Further investigation shows----->
2. Clients connected to 'access-SWITCH01' can connect; but...
3. Clients connected to 'access-SWITCH02' fail to connect
4. Clients on both access-SWITCHes can connect to web traffic on a different server (same subnet) that is not NLB-clustered
5. The servers are on an ESXi host connected to the 'distribution-Layer3-SWITCH', which in turn connects to both access-SWITCHes.
0
 
page1985Commented:
Are you using VMware Distributed Switching or standalone vSwitches?
0
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

 
page1985Commented:
Also, are the host adapters teamed (belong to the same vSwitch or dvSwitch)?
0
 
waltforbesSenior IT SpecialistAuthor Commented:
Hi page1985:
1. Standalone vSwitches are being used
2. There is only 1 (one) host adapter in the vSwitch being used by the virtual machines
0
 
page1985Commented:
Is promiscuous mode set to "Accept" in the properties of the vSwitch?
0
 
waltforbesSenior IT SpecialistAuthor Commented:
It is set to "Reject". What does this mode determine? How is it relevant to NLB clustering?
0
 
waltforbesSenior IT SpecialistAuthor Commented:
Hi page1985:
The nodes of the NLB cluster are accessible (http) by all clients - failure occurs only when using the cluster name from 'access-SWITCH02' workstations.
0
 
page1985Commented:
It is relevant to NLB clustering because "promiscuous mode" is what determines whether packets with forged MAC addresses are permitted on the interface.  NLB clustering, as you will notice, forges the MAC address of the network interfaces it belongs to.  If you do not allow "promiscuous mode" (also called "forged transmit"), then NLB clustering may not work at all or may work unpredictably.
0
 
waltforbesSenior IT SpecialistAuthor Commented:
To Page1985:
When I configured the vSwitch to accept promiscuous mode, the ESXi host issues the following alarm: "Network uplink redundancy lost". What should I do? What will be the consequences?
0
 
page1985Commented:
That means that you have only one physical network adapter in the vSwitch.
0
 
page1985Commented:
For redundancy lost, here's a link:
http://communities.vmware.com/thread/252066

For configuring NLB on VMware ESXi, here's another link:
http://www.vmware.com/files/pdf/implmenting_ms_network_load_balancing.pdf

Additional information for Multicast mode:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006558

Challenges with Unicast mode:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1556

Specifically, NLB in "Unicast" mode requires special configuration.  However, if you would like to disable "promiscuous mode", you can configure Windows NLB to use Multicast mode instead.  This requires no special configuration within ESXi.  Additionally, this is the recommended configuration for NLB clusters that only have one network adapter because Unicast mode breaks the ability for nodes in the cluster to communicate with each other.

Another major point behind the functionality of Unicast mode NLB that is worth evaluating -- unless your physical network switces are not multicast aware (most modern siwtches are), operating in Unicast mode is ill advised because unicast mode works by masking the MAC address of the hosts (hence why we need forged transmit) in order to prevent your network hardware from identifying the hosts.  This results in all traffic destined for the NLB cluster to be blasted to every node on the network.  This, obviously, creates network overhead and is all around a situation we want to avoid if we can.
0
 
waltforbesSenior IT SpecialistAuthor Commented:
Hi Page1985:
Wonderful information! Please note the following:
1. The NLB nodes are on different ESXi hosts
2. Action Completed - I configured the vSwitches on each ESXi host to accept promiscuous mode.
3. Result: no success.
QUESTION: Is it mandatory that both NLB nodes be on the same ESXi host?
0
 
page1985Commented:
If you use Unicast mode, yes, both VMs must be on the same host.  You can disable promiscuous mode (basically go back to the way you were before we started), then configure Windows NLB to use Multicast mode and you should be good that way.

I believe Unicast vs Multicast in Windows NLB manager is just a toggle, so you don't have to destroy and recreate the cluster to switch.  To do this:

Open Windows Network Load Balancing
Once the cluster loads, right click the cluster itself (not a node) and select Cluster Properties
Click the Cluster Parameters tab
Select the Multicast bubble, then click OK

The NLB cluster will reconfigure itself and after a minute or so you should see all nodes go yellow (converging), and then with some F5/refresh, eventually go green.
0
 
page1985Commented:
As a side note (not really related to this incident), it is also recommended that you configure the port rules to the specific service ports (such as 80 and 443) that are actually being load balanced. Certain protocols, like RPC, may have issues if someone tries to connect to them at the cluster level instead of the node level and this prevents connections from being made to non-balanced services via the balanced IP.  This is, however, not important to your current objective and for a web server shouldn't really be a big deal.
0
 
waltforbesSenior IT SpecialistAuthor Commented:
Hi Page1985:
1. I changed the cluster operation mode to multicast: this caused cluster to fail - icmp (ping) traffic could not reach the cluster IP address - not even from a host in the same subnet.
2. I do agree that the port rules should be configured for specific service ports.
0
 
page1985Commented:
OK.   What is the make/model of the switch?  I'll check to see if it supports multicast.
0
 
waltforbesSenior IT SpecialistAuthor Commented:
Hi Page1985:
The switch model is a Cisco Catalyst 6506E.
0
 
page1985Commented:
Ok.  The Cisco Catalyst 6500 series switches requires static ARP registration when placed in multicast mode.  Here's Cisco's writeup on it:

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml

Additionally, here is a third party article on Cisco and Windows NLB:
http://inetpro.org/wiki/Configuring_Cisco_to_work_with_a_Windows_NLB_Cluster
0
 
waltforbesSenior IT SpecialistAuthor Commented:
Hi Page1985,
you have been so expertly impressive, resourceful, patient and so helpful. My understanding is much expanded, and my issue resolved! Many thanks for your great service!
0
 
page1985Commented:
No problem.  Glad I could help you out.  I remember setting up my own clustering not too terribly long ago and feeling just as ready to pull out my hair.  :)
0

Featured Post

Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

  • 12
  • 9
Tackle projects and never again get stuck behind a technical roadblock.
Join Now