Multiple IP Address on a single teamed NIC
We are in the process of configuring a windows 2008 server, which will host an apache application and an IIS application, under different DNS names. The simplest approach we could think of was to apply two separate IP addresses, bind one to the apache instance, and bind one to the IIS instance (using the netsh ip listen command).
To do this we have teamed the networks cards (Broadcom 5709 in a Dell R610 server) together using the BACS3 suite - all good so far. Next we added the static IP for the server, then added an additional IP address to the card. All seems to be fine.
THEN - all of a sudden other sites are unable to connect to the second IP address, then the primary one drops as well. Strangely, if I am connected to the server through RDP I can continue to access the machine without issue, but the machine then reports "no internet access" and i am unable to ping anything (even the default router). This problem disappears after a reboot, but then comes back again shortly after.
If i change the secondary Ip address everything works again, then stops working again shortly after!
I suspect this is down to some sort of weird routing issue (latest network drivers and management software from Broadcom already tried - as has a replacement mainboard from Dell!) - but can't fathom out why we have a similar setup elsewhere (two servers with teamed nics, running multiple IP addresses, but with MS Load balancing) - and these work faultlessly?
To do this we have teamed the networks cards (Broadcom 5709 in a Dell R610 server) together using the BACS3 suite - all good so far. Next we added the static IP for the server, then added an additional IP address to the card. All seems to be fine.
THEN - all of a sudden other sites are unable to connect to the second IP address, then the primary one drops as well. Strangely, if I am connected to the server through RDP I can continue to access the machine without issue, but the machine then reports "no internet access" and i am unable to ping anything (even the default router). This problem disappears after a reboot, but then comes back again shortly after.
If i change the secondary Ip address everything works again, then stops working again shortly after!
I suspect this is down to some sort of weird routing issue (latest network drivers and management software from Broadcom already tried - as has a replacement mainboard from Dell!) - but can't fathom out why we have a similar setup elsewhere (two servers with teamed nics, running multiple IP addresses, but with MS Load balancing) - and these work faultlessly?
ASKER
Hi LesterClayton, thanks for the response. I've currently tried this in the following configurations:
- all four interfaces teamed in Smart Load Balancing mode, all cables, 2 cables, 1 cable connected to different ports in all possible combinations.
- 2 interfaces teamed together - same as above to test
- 1 interface in team
- single nic (no teaming and no teaming software installed). both ip addresses on same card
- 2 separate nics, one ip on each nic - THIS IS THE ONLY ONE THAT WORKS!
I have also installed the latest firmware update for the network from Dell, and tried both the dell latest drivers, and BACS3 software, and the latest drivers from Broadcom's website, with the BACS4 software. No change.
With regards to switch hardware, we are running HP Procurve 5400zl series, so no issues there - we also have other R610 servers on the same network using the smart laod balancing option and they are all running perfectly.
I have even changed the network cables, and swapped ports on the switch to a different module!
- all four interfaces teamed in Smart Load Balancing mode, all cables, 2 cables, 1 cable connected to different ports in all possible combinations.
- 2 interfaces teamed together - same as above to test
- 1 interface in team
- single nic (no teaming and no teaming software installed). both ip addresses on same card
- 2 separate nics, one ip on each nic - THIS IS THE ONLY ONE THAT WORKS!
I have also installed the latest firmware update for the network from Dell, and tried both the dell latest drivers, and BACS3 software, and the latest drivers from Broadcom's website, with the BACS4 software. No change.
With regards to switch hardware, we are running HP Procurve 5400zl series, so no issues there - we also have other R610 servers on the same network using the smart laod balancing option and they are all running perfectly.
I have even changed the network cables, and swapped ports on the switch to a different module!
This is a most unusual problem indeed - I'm afraid I cannot offer any more suggestions. I run a few servers with multiple IP's on the same NIC and have never come across this particular issue, but admittedly I do not have a single server which runs both apache and IIS - it's either one or another. I can't think that this could be the cause of the issue.
More troubleshooting you can do - when this problem occurs, and you type NETSTAT -AB on the server - do you still see the processes listening on the various IP's and ports? Can you try to telnet to the IP address and port number (80) on the LOCAL server? This would check if the IP Stack is still listening without going through the network.
Also, when you do the NETSTAT do you get lots of TIME_WAIT or CLOSE_WAIT statusses? What might be happening is that when connections are being made and closed, they are not being freed up properly from the IP stack, and you're effectively running out of ports it can use.
More troubleshooting you can do - when this problem occurs, and you type NETSTAT -AB on the server - do you still see the processes listening on the various IP's and ports? Can you try to telnet to the IP address and port number (80) on the LOCAL server? This would check if the IP Stack is still listening without going through the network.
Also, when you do the NETSTAT do you get lots of TIME_WAIT or CLOSE_WAIT statusses? What might be happening is that when connections are being made and closed, they are not being freed up properly from the IP stack, and you're effectively running out of ports it can use.
ASKER
thanks - i will try this as soon as i can take the server out of service again to test!
Perhaps you should do a NETSTAT -AB now to see what it looks like, it would be a good to do a baseline comparison against the other test. If you get too many results, you can naturally pipe it to a file.
Maybe this will help, check out this http://blogs.technet.com/b/josebda/archive/2010/09/03/using-the-multiple-nics-of-your-file-server-running-windows-server-2008-and-2008-r2.aspx
I am going to presume that all of the addresses are in the same subnet.
I have multiple HP servers with multiple IP addresses on tow and four port teamed NICs connected to 5412zl switches and have not experienced what you have described.
In your Network Teaming configuration, did you specify a specific MAC address to be used on the TEAM connection ?
What trunk configuration did you do on the Switch ?
Do you have any port security setup on your switch and/or firewall that could be limiting traffic from multiple addresses on one MAC address or multiple MAC addresses on one switch port (an LACP trunk is effectively a single switch port).
I have multiple HP servers with multiple IP addresses on tow and four port teamed NICs connected to 5412zl switches and have not experienced what you have described.
In your Network Teaming configuration, did you specify a specific MAC address to be used on the TEAM connection ?
What trunk configuration did you do on the Switch ?
Do you have any port security setup on your switch and/or firewall that could be limiting traffic from multiple addresses on one MAC address or multiple MAC addresses on one switch port (an LACP trunk is effectively a single switch port).
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
issue is due to a limitation in the hardware on certain Dell servers
If you don't get the issue, then look at your load balancing mode, and see if it's supported on the switch.
If you do get the issue, remove the team, and then just on one adapter, bind the two IP addresses (leave the other adapter disabled or something). Test.
These will help you rule out various things, and help you conclude what the root cause of the problem is.