ISA2006 Unicast NLB not working

This issue has been going on for sometimes. I have called up Microsoft and they have been unable or I should say not able to resolve the issue %100. Here is the case in brief explenation:

2 Servers, DL380G4
3 Cards in each (A=LAN, B=External, C=Intra-Array) identical on both servers
Template used = Back-Firewall
One defaultgateway address on External NIC both servers
Routing table updated with static ip range to cover the entire network

Both servers connected to a dummy Hub and from the Hub to the Layer 3 switch as per Microsoft Article:

as well

Nothing goes outbound. The Microsoft Firewall services shutsdown and of course there is no communication between ADAM and A.D. Has anybody seen this or has found a solution to correct the connectivity? Any guidance would help


Who is Participating?
Keith AlabasterConnect With a Mentor Enterprise ArchitectCommented:
Thanks :)

Keith AlabasterEnterprise ArchitectCommented:
So are you using the NLB from the clustering/nlb services or the NLB that comes directly as part of ISA 2006 server in the Enterprise version?

I assume you are notr splitting tyhe inbound and the outbound traffic as it is ISA server.

You say you are seeing the fw service shutting down? What messages?
How is the CSS setup? One on each server? A central location?

What is the OS you are running? What service pack?
What is the network card in the servers - Broadcoms or Intels?

jabramoAuthor Commented:
Hi Keith,

Thanks for getting back to this.

We're not using the ISA server as a FW service as we have a hardware FW installed that is servicing the organization. So, there is only Enterprise policies in places.

There are two CSS servers one on each server

Microsoft FW stops responding when checking the Services and Event Viewer. I don't recall what the message is but it is a general service shutdown message.

The OS is Windows 2003 Enterprise Edition with SP1

The network cards in the servers are HPNC7782 bult-in cards from HP. They are not teamed. That I know for sure.
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

Keith AlabasterEnterprise ArchitectCommented:
Sorry if i am being slow but if you are using internal/external cards, doesn't that still make ISA a firewall? It may just be different terminology that is confusing me. What sort of things are you putting just in the Enterprise policy? I have over 200 ISA installs behind me now but have not seen that configuration as far as I know.

Did you get anything out of the BPA?  You need .net 1.1 for this.
jabramoAuthor Commented:
Well what I meant is that there is no Firewall policies access rules. Only Enterprise policy which was created based on Organizational needs. The Enterprise policies covers, restricted web site access, FTP usage, blocked web sites to certain users and groups and etc...

The problem I'm facing is the Layer 3 switch. It doesn't allow packets to go through. We discussed this with Microsoft and Nortel Networks (Nortel Switch Passport 8600) and they all said to put a dummy hib to have one MAC address sent to the switch. We cleared wht ARP entry and still no sign of ARP caching on the switch.

I have done number of ISA installation back from Proxy 2.0 all the way to ISA 2006 and as you said, this one must have been one of a kind.
Keith AlabasterEnterprise ArchitectCommented:
OK - understood.

I can see that the reasoning behind the dummy hub is then the switch only sees the MAC coming from the one switch-port even though in reality there are actually two sources for it.

Can you SPAN the switchport that the ISA'a are coming in on? What do you see of you run up wireshark or netmon 3.1?
Is the fw service dying straight away?
Does the Nortel feed out to a syslog server at all? Any debug info?

jabramoAuthor Commented:
Wireshark see the packets are leaving the source MAC address and Destination MAC. There is no issue with the wireshark. When I ping the internal gateway I get no response at all but wire shark saus it has reached the destination. The PING access rule is also created to allow ICMP requests.

The FW services dies few seconds after NLB is created and services want to start. I'm not sure of Nortel feeds out to Syslog. That is somebody else is looking after this.
Keith AlabasterEnterprise ArchitectCommented:
And ICMP is enabled on the ISA system policies as well?

If you disconnect the Nortel and put a plain old PC in its place, are you getting the same response/conditions?

jabramoAuthor Commented:
The ICMP is enabled on the ISA system.

I can't replace the Nortel switch since it is the core switch.
Keith AlabasterEnterprise ArchitectCommented:
No, I wasn't suggesting replacing the Nortel - they are good boxes. However, out of hours, I was wondering if you could connect a PC to the hub instead of connecting the hub to the Nortel - this would remove the layer 3 element out of the equation altogether.

The fact that you have ICMP enable in both places (the system policy and in the firewall policy) but do not get a return is so bizarre. I assume it is because the fw service has stopped.

Not sure what you have been through with Microsoft but to me (at least) the next steps would be to backup the ISA configs and then remove ISA server. ie Does the system function with NLB correctly at the operating system level. If it doesn't work there then ISA is simply confusing the issue.

jabramoAuthor Commented:
We have applied all the options. We have added a desktop server to the hub as you suggested. The Windows based NLB works fine without the ISA server which was one of the test we needed to make sure the NLB actually functions. No problem there. We re-imaged the servers with OS only and services pack 1 and then added ISA 2006 in array model without a problem. As soon as we enable Integrated NLB, BAMMMMM, that's where it fails. So at ths point we assume this is the NLB issue but rather than ISA. Never seen ISA to spit out so many issues involving NLB.

Running out of ideas so as Microsoft. I was on the phone with MS Premier support for the total of 3 days and nothing could be resolved. So there it is.

Keith AlabasterEnterprise ArchitectCommented:
And the best practice analyser doesn't show any anomalies either?

I have a spare box here so will build this up and see if i can reproduce the symptoms. Sounds interesting.
jabramoAuthor Commented:
Microsoft suggested to have the MBPA 5.0 be installed on the server. I ran the MBPA but there was nothing tp report. That's where I find it odd since if MBPA doesn't say anyting then how the heck can we trace down the issue?
BTW i just downloaded a doc from MS and it says:

Switch Flooding
Masking the cluster media access control (MAC) address on outgoing packets prevents the switch from associating the cluster MAC address with a single port. When a client request (which contains the cluster MAC address) enters the switch, the switch does not recognize the MAC address in the packet and so sends the packet to all ports. This is called switch flooding.
In unicast mode (in which ISA Server integrated NLB operates), NLB induces switch flooding by design, so that packets sent to the cluster's virtual IP address go to all the cluster hosts. Switch flooding is part of the NLB strategy of obtaining the best throughput for any specific load of client requests.
If, however, the cluster shares the switch with other (noncluster) computers or other clusters, switch flooding can add to the other computers' network overhead by including them in the flooding.
You can avoid flooding noncluster computers by putting a network hub between the switch and the NLB cluster hosts, and then disabling the MaskSourceMAC feature. The hub delivers each packet to every host, and the switch associates the cluster MAC address with a single port, satisfying the switch's requirement that each port be associated with a unique MAC address.
To disable the MaskSourceMAC feature, follow this procedure on each member of the ISA Server array, after you configure integrated NLB on the ISA Server computer. At a command prompt, type the following commands:      
1.      nlb registry masksrcmac off
2.      nlb reload

I did all those except #1

Would that be possible?
create a firewall policy on the top of all policies that allow all traffic to pass between your array isa servers and see.

i.e: Array member server rule>Allow>all outbound traffic>from array servers>to all networks>all users

try and let us know.
jabramoAuthor Commented:
Thanks for the reply.

We're almost there. I found a 61 page document from Nortel Corporation that describes How to configure Microsoft NLB on Nortel Passport 8600. The 61 page document explains the exact topology and different configuration methods as well as requirements to have the latest update 3.7.14/4.1.1 in order to have the Unicast function as it should.

The results and conclusion will be posted upon testing completion.

Thank you
Keith AlabasterEnterprise ArchitectCommented:
Well done!  :)

Would be useful if you could post a link to the doco you found as well.
jabramoAuthor Commented:

Here it is:

We use 8600. Other configuration models available
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.