We help IT Professionals succeed at work.

Multi-homed Win2012R2 server passing private NIC traffic out production NIC gateway. Why?

1,029 Views
1 Endorsement
Last Modified: 2017-08-15
I'm seeing a behavior on Win2012R2 that's different from Win2003R2, on how routing works on multi-homed servers that don't have routing turned on. I want the Win2012R2 server to behave like Win2003R2.

Win2012R2 server
NIC1_Production
172.18.138.60
255.255.255.0
GW 172.18.138.254

NIC2_Private
172.20.0.1
255.255.0.0
no GW

I have 172.16.0.0/24 devices on my NIC2_Private network. But on the production side of my network (on the NIC1_Production side), several router hops away across my MPLS I also have 172.16.0.0/24 devices. I want this 2012R2 server to only talk to the private 172.20.0.0/16 network, which is why I multi-homed a server without routing turned on.

The problem: On the 2012R2 server, when I ping devices in the 172.20.0.0/16 range that exist on my production network (but not on my private network), the pings try NIC2_Private for the first attempt, but then go out NIC1_Production's gateway (my production network) for the rest. Why, if the 172.20.0.0 is a network directly connected to the server? In contrast, on a Win2003R2 server, the pings only stay on NIC2_Private and don't go out NIC1_Production. How do I change this 2012R2 behavior so it acts like Win2003R2?

I have a Win2003R2 multi-homed server connected to both networks also, and it doesn't pass the private traffic to the production network (which is what I want).

As of now, I don't have any devices on both networks with the same IP address. That may change at some point in the future, which is why I want to fix this problem now.

Here's my 2012R2 server test results:

C:\Users\administrator.VALUEPLASTICS>ping 172.20.1.4

Pinging 172.20.1.4 with 32 bytes of data:
Reply from 172.20.0.1: Destination host unreachable.
Reply from 172.20.1.4: bytes=32 time=47ms TTL=121
Reply from 172.20.1.4: bytes=32 time=62ms TTL=121
Reply from 172.20.1.4: bytes=32 time=48ms TTL=121

Ping statistics for 172.20.1.4:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 47ms, Maximum = 62ms, Average = 52ms


Here's a trace:
C:\Users\administrator.mydomain>tracert -d 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1    <1 ms     *       <1 ms  172.18.138.254
  2    <1 ms    <1 ms    <1 ms  172.18.138.253
  3     4 ms    44 ms     8 ms  100.65.0.37
  4    34 ms    34 ms    34 ms  100.65.0.13
  5    36 ms    36 ms    36 ms  100.65.0.14
  6    36 ms    37 ms    36 ms  172.18.10.241
  7    36 ms    36 ms    36 ms  172.18.10.240
  8     *        *        *     Request timed out.
  9    66 ms    60 ms    72 ms  172.20.1.4

Trace complete.

Here's my IP config:
C:\Users\administrator.mydomain>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : myserver1
   Primary Dns Suffix  . . . . . . . : mydomain.com
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : mydomain.com

Ethernet adapter NIC1_Production:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet #6
   Physical Address. . . . . . . . . : B0-83-FE-C1-98-27
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.18.138.60(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 172.18.138.254
   DNS Servers . . . . . . . . . . . : 172.18.138.4
                                       172.18.138.21
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter NIC2_Private:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet
   Physical Address. . . . . . . . . : B0-83-FE-C1-98-28
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.20.0.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . :
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{4C9E1AFE-99BD-4617-A060-C1FECFA6BE31}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{F133CFC7-B7FD-4459-AD0D-B3CD1C4B3203}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes


Here's my routing table:
C:\Users\administrator.mydomain>route print
===========================================================================
Interface List
 19...b0 83 fe c1 98 27 ......Broadcom NetXtreme Gigabit Ethernet #6
 14...b0 83 fe c1 98 28 ......Broadcom NetXtreme Gigabit Ethernet
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 13...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface               Metric
          0.0.0.0                           0.0.0.0   172.18.138.254    172.18.138.60    266
        127.0.0.0                    255.0.0.0         On-link                     127.0.0.1    306
        127.0.0.1       255.255.255.255         On-link                     127.0.0.1    306
  127.255.255.255  255.255.255.255         On-link                   127.0.0.1    306
     172.18.138.0        255.255.255.0         On-link            172.18.138.60    266
    172.18.138.60   255.255.255.255         On-link            172.18.138.60    266
   172.18.138.255  255.255.255.255         On-link            172.18.138.60    266
       172.20.0.0               255.255.0.0         On-link                   172.20.0.1    266
       172.20.0.1      255.255.255.255         On-link                   172.20.0.1    266
   172.20.255.255  255.255.255.255         On-link                  172.20.0.1    266
        224.0.0.0                     240.0.0.0         On-link                    127.0.0.1    306
        224.0.0.0                     240.0.0.0         On-link           172.18.138.60    266
        224.0.0.0                     240.0.0.0         On-link                  172.20.0.1    266
  255.255.255.255  255.255.255.255         On-link                   127.0.0.1    306
  255.255.255.255  255.255.255.255         On-link          172.18.138.60    266
  255.255.255.255  255.255.255.255         On-link                 172.20.0.1    266
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address      Metric
          0.0.0.0                          0.0.0.0     172.18.138.254        Default
===========================================================================

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None


I read some articles about strong and weak host models, such as https://technet.microsoft.com/en-us/library/2007.09.cableguy.aspx. I tried setting these commands on both NIC's, but it didn't make any difference:
• netsh interface ipv4 set interface NIC1_Production weakhostsend=enabled
• netsh interface ipv4 set interface NIC1_Production weakhostreceive=enabled
• netsh interface ipv4 set interface NIC2_Private weakhostsend=enabled
• netsh interface ipv4 set interface NIC2_Private weakhostreceive=enabled
 


For comparison, here's my Win2003R2 server multi-homed on the same two networks...

C:\Documents and Settings\administrator>ping 172.20.1.4

Pinging 172.20.1.4 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 172.20.1.4:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),


C:\Documents and Settings\administrator>tracert -d 172.20.0.109

Tracing route to 172.20.0.109 over a maximum of 30 hops

  1     *        *        *     Request timed out.
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.


C:\Documents and Settings\administrator>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : myserver2
   Primary Dns Suffix  . . . . . . . : mynetwork.com
   Node Type . . . . . . . . . . . . : Unknown
   IP Routing Enabled. . . . . . . . : Yes
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : mynetwork.com

Ethernet adapter NIC1_Production:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : vmxnet3 Ethernet Adapter #2
   Physical Address. . . . . . . . . : 00-50-56-AA-00-0D
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 172.18.138.8
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 172.18.138.254
   DNS Servers . . . . . . . . . . . : 172.18.138.4
                                       172.18.138.21

Ethernet adapter NIC2_Private Network:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : vmxnet3 Ethernet Adapter
   Physical Address. . . . . . . . . : 00-50-56-AA-00-12
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 172.20.75.1
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . :


C:\Documents and Settings\administrator>route print

IPv4 Route Table
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x10003 ...00 50 56 aa 00 0d ...... vmxnet3 Ethernet Adapter #2
0x10004 ...00 50 56 aa 00 12 ...... vmxnet3 Ethernet Adapter
===========================================================================
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway                    Interface      Metric
          0.0.0.0                            0.0.0.0   172.18.138.254        172.18.138.8     10
        127.0.0.0                     255.0.0.0              127.0.0.1               127.0.0.1      1
     172.18.138.0         255.255.255.0       172.18.138.8        172.18.138.8     10
     172.18.138.8     255.255.255.255             127.0.0.1               127.0.0.1     10
   172.18.255.255  255.255.255.255       172.18.138.8        172.18.138.8     10
       172.20.0.0               255.255.0.0          172.20.75.1          172.20.75.1     10
      172.20.75.1      255.255.255.255             127.0.0.1               127.0.0.1     10
   172.20.255.255  255.255.255.255         172.20.75.1           172.20.75.1     10
        224.0.0.0                     240.0.0.0       172.18.138.8         172.18.138.8     10
        224.0.0.0        240.0.0.0      172.20.75.1      172.20.75.1     10
  255.255.255.255  255.255.255.255     172.18.138.8     172.18.138.8      1
  255.255.255.255  255.255.255.255      172.20.75.1      172.20.75.1      1
Default Gateway:    172.18.138.254
===========================================================================
Persistent Routes:
  None

Any ideas?

Thanks,
Jeff
Comment
Watch Question

CERTIFIED EXPERT
Distinguished Expert 2018
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION

Author

Commented:
I opened a case with Microsoft on June 8, 2017 for this. They haven't been able to offer a solution to make the private NIC work like it did in Win2003. As of today (July 14, 2017), I still have no solution for this issue.

Here are the main comments from the ticket, in case they're of interest to anyone.
-------------------------------------------------------------------------

Hi Jeff,

We reviewed the logs and we believe that what you are seeing is expected behavior.
This behavior was introduced as part of the NUD (Neighbor Unreachability Detection) integration into the stack with Windows Server 2008 and later.

Below is an extract from our blog.
<Quote>
If you have a specific route defined and ARP connectivity to that remote router/gateway fails. For example, in Windows XP if you tried to ping the remote router/gateway and ARP failed to get a response, you would receive “destination host unreachable” and it would stop. After Windows Vista, if you performed the same action and ARP fails to get a response, Windows will mark the address as unreachable and look for another route. If no other specific route is found it will use the default gateway instead.  Assuming ARP works to the default gateway, it would then try ICMP to the remote resource and you would either get a reply or a time-out depending on whether the resource is reachable across the default gateway or not.
</Quote> Ref: https://blogs.technet.microsoft.com/networking/2009/04/24/source-ip-address-selection-on-a-multi-homed-windows-computer/

We would like for you to run the following command and see if the desired is achieved.
•      netsh interface ipv4 set interface 14 nud=disabled

The above command will disable the NUD functionality which should prevent the machine from reaching out to the end resource via alternate route.

Please could you test this and let me know the results, if the behavior persists, can you please collect the output of below commands:
•      arp -a
•      route print
•      netsh trace start scenario=netconnection capture=yes report=yes
•      netsh trace stop
•      Upload the ETL file
---------------------------------------------------------------------------------------------------------------

Jeff to Microsoft engineer: That command doesn't change anything. The problem is still occurring.

---------------------------------------------------------------------------------------------------------------

Jeff,

I have been doing further research and testing in my lab environment, also tested running that command and even rebooting the server, and the interface still doesn’t change the Neighbor Unreachability Detection (NUD) setting, although it doesn’t give any error messages.

I reached out to some colleagues regarding this and I need to get back to you with their findings; so please hold on scheduling the maintenance window until I can get a definite action plan that will allow us to change that setting on the server.
---------------------------------------------------------------------------------------------------------------------------------------------------

Hello Jeff,

I wanted to reach out to you and let you know where we are with this issue:

After version 2008, there were changes made to the operation system because of the RFC for IPv6. As we went from dual-stack to dual-protocol, some IPv6 functionality was merged from IPv6 to IPv4, and one of these was NUD (Neighbor Unreachability Detection). Basically, if ARP fails, or if the traffic fails going out, Windows will try any and all available routes by default. This is what is causing the traffic to go from the Private NIC to the Production NIC.

There is the command to turn it off (netsh interface ipv4 set interface 14 nud=disabled), as we have tested, but it is not being honored. In fact, after having involved several levels internally, the reason the ‘show interface’ never showed NUD as disabled was because the command was not being enforced. We have sent this issue, and all we have found to a group that includes the Product Group and are waiting to see if they respond. If you want and need to, we can also create a design change request, but we are not sure of the timeframe.

As you had requested before, there is another option for a workaround, that doesn’t involve having the Firewall turned on, it’s to build a legacy IPSec rule and to implement it on the server. Please be aware that the Microsoft guidance is to not touch legacy IPSec in versions 2008 and later. Implementing this will be an overhead.

I can also give you a call to discuss this further if needed, but I wanted you to have this information in writing in case you need to involve Management. Please let me know if you have any further questions and how would you like to proceed with the case.
--------------------------------------------------------

Hi Jeff,

I hope you’ve had a good weekend. After more research and additional testing, it seems that the command we discussed before does perform the change, but the output of “netsh int ipv4 show interface” has a cosmetic error. If you perform an interface dump, the changes show applied.

I tested this on two lab servers and the dump shows the changes applied. So if you can please proceed with the steps:
1.      On an Admin session of Powershell, run:
a.      set-NetIPInterface -InterfaceIndex 14 -NeighborUnreachabilityDetection Disable -PolicyStore PersistentStore
b.     disable the NIC, re-enable the NIC      
c.     netsh -c interface dump > c:\netsh.txt
2.      Please collect the output of below commands:
a.      arp -a
b.      route print
c.      netsh trace start scenario=netconnection capture=yes report=yes
3.      Reproduce the issue
a.      netsh trace stop
4.      Upload the ETL file

Please go ahead and coordinate when the maintenance window can be scheduled, and let me know.
--------------------------------------------------------------------------------------------------------------------------------

Jeff to support engineer: The problem still exists.
--------------------------------------------------------------------------------------------------------------------------------

Hi Jeff,

I’m sorry to hear that didn’t resolve the current issue
-------------------------------------------------------------------------------------------------------------------------------
Hi Jeff,

So we have been able to reproduce your issue in our labs, and have been seeing the same behavior. We have put questions out to our team, but haven’t received a response yet.

However, there is a way that we can keep the traffic segregated as a time sensitive solution: to create a rule in the Windows Firewall to block traffic from the Private network to go out of the Production network assigned NIC, this makes the PING in our environments fails with a “generic failure”.

The steps to configure the rule are:
•      Open the Windows Firewall Console
•      Select Outbound Rules> New Rule> Custom> All programs> Ports/Protocols: Any
•      Which local IP addresses does this rule apply to? Here you need to set the IP of the interface you don’t want to forward traffic: the Production NIC IP.
Which remote IP addresses does this rule apply to? Here you need to set the Network IP of the Private subnet (172.20.0.0).
•      Block Connection> Select all networks> Name rule and save

After this you can test if the traffic is allowed. If we do hear something from our team we will let you know, but we wanted to give you an alternative to have your environment working.
-------------------------------------------------------------------------------------------------------------------------

Jeff to support engineer: We don't have the firewall enabled on this server and don't want to for various business reason.
-------------------------------------------------------------------------------------------------------------------------

Hi Jeff,

Thank you for letting me know about the firewall option not working for your environment. I will continue with the research to get you other options, and also push forward the research on the server behavior.
---------------------------------------------------------------------------------------------------------------------------

Hi Jeff,

I’m pending an update from the product group. I will check in with them and see if I can get anything for you before the end of the week.
------------------------------------------------------------------------------------------------------------------------------------------

Hello Jeff,

Sorry for the delayed update. I want to follow up and let you know that we have had no response from the PG as of yet. There were more resources involved for visibility but we are still pending on their side to come back with a response.

I am contacting some additional resources to try to push a bit on our side and see if we can get any information.
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION

Author

Commented:
Was the best solution.
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.