WorkSoft
asked on
Duplicate IP Detected on any machine, for any public ip, virtual or not
O.k. this is a good one. Pull up an office chair and slam down a Mountain Dew, you are going to want to focus.
We have 2 datacenters, HQ & Colo. Details below:
HQ:
30MB Internet from ProviderB (55.55.55.5/30 running BGP with a class C (99.99.99.0/24) pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.2), connected with an Adva (?) device on both ends.
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.101.0/24
10.1.102.0/24
10.1.103.0/24
10.1.104.0/24
These go into a big stack of Dell Powerconnects and a large VMWare Vcenter 5 farm.
(Migrating away from a 30MB non-BGP connection (22.22.22.2/27) that passes a /27 to me in bridged mode for publicly IPed servers which come straight from the 2816 to another unmanaged 2816 that then runs a network connection a virtual nic on each ESX host)
Colo:
30MB Internet from ProviderA (66.66.66.6/30 running BGP with a class C pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.3)
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.111.0/24
10.1.112.0/24
10.1.113.0/24
10.1.114.0/24
These go into a stack of Dell Powerconnects and a small but important VMWare Vcenter 5 farm.
The point to points are set up to route anything from the opposing network to the point to point gateway on the other side and all routes have been configured.
Everything up to now runs like a dream, routing works on both sides, everything talks fine.
We then went to move MS Lync from the HQ and move it to the Colo by setting up a NAT of 99.99.99.10 to 10.1.112.10 and it fails. (We could cover Lync and NAT for days and we would get nowhere, that problem is for another day.) Long story short the only option is you can't NAT to a Lync Server that is handling incoming SIP requests so we call ProviderA and ask for 1 IP. They agree and we get 88.88.88.8 assigned to us. We assign it to the virtual Lync 2013 server and immediately get a Duplicate IP address error. We check the windows logs and it says it's due to the MAC 64-D8-14-20-36-C2 has it. That is weird since we've never used 88.88.88.8. We checked all ARP tables on ALL switches, firewalls, VMs, ESXi hosts and can't find it at all. We tried a brand new server that had just been built and we got the same error, same MAC. We plug a laptop into the unmanaged 2816 and try that IP on it, Same error, same MAC. We swap out the 2816 for a BRAND NEW 2816 plug the laptop in and get the same error, same MAC.
We plug the same laptop into the ethernet hand-off ProviderA gives us and it works like a charm. No issues.
So we ruled out VMware, the 2816, the laptop we used to test and ProviderA. So we figure it must 88.88.88.8. I have the old /27 that we are migrating from and I ask ProviderA to move it from the HQ circuit to the ColoCircuit and they do. I assign an IP address that has never been used to the server and I get, wait for it, the same error, same MAC address. I then assign another IP from the block, same error, same MAC, another IP and I get the same error, the same MAC. I tried 6 IPs and they all gave me the same result. Oh and before I moved that IP Block, I physically and personally removed it from all the servers it did reside on and flushed all ARP caches in my network and waited 30 minutes. (That was 4 hours ago and I just tested it and still the same results.)
Whiskey Tango Foxtrot...
We have 2 datacenters, HQ & Colo. Details below:
HQ:
30MB Internet from ProviderB (55.55.55.5/30 running BGP with a class C (99.99.99.0/24) pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.2), connected with an Adva (?) device on both ends.
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.101.0/24
10.1.102.0/24
10.1.103.0/24
10.1.104.0/24
These go into a big stack of Dell Powerconnects and a large VMWare Vcenter 5 farm.
(Migrating away from a 30MB non-BGP connection (22.22.22.2/27) that passes a /27 to me in bridged mode for publicly IPed servers which come straight from the 2816 to another unmanaged 2816 that then runs a network connection a virtual nic on each ESX host)
Colo:
30MB Internet from ProviderA (66.66.66.6/30 running BGP with a class C pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.3)
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.111.0/24
10.1.112.0/24
10.1.113.0/24
10.1.114.0/24
These go into a stack of Dell Powerconnects and a small but important VMWare Vcenter 5 farm.
The point to points are set up to route anything from the opposing network to the point to point gateway on the other side and all routes have been configured.
Everything up to now runs like a dream, routing works on both sides, everything talks fine.
We then went to move MS Lync from the HQ and move it to the Colo by setting up a NAT of 99.99.99.10 to 10.1.112.10 and it fails. (We could cover Lync and NAT for days and we would get nowhere, that problem is for another day.) Long story short the only option is you can't NAT to a Lync Server that is handling incoming SIP requests so we call ProviderA and ask for 1 IP. They agree and we get 88.88.88.8 assigned to us. We assign it to the virtual Lync 2013 server and immediately get a Duplicate IP address error. We check the windows logs and it says it's due to the MAC 64-D8-14-20-36-C2 has it. That is weird since we've never used 88.88.88.8. We checked all ARP tables on ALL switches, firewalls, VMs, ESXi hosts and can't find it at all. We tried a brand new server that had just been built and we got the same error, same MAC. We plug a laptop into the unmanaged 2816 and try that IP on it, Same error, same MAC. We swap out the 2816 for a BRAND NEW 2816 plug the laptop in and get the same error, same MAC.
We plug the same laptop into the ethernet hand-off ProviderA gives us and it works like a charm. No issues.
So we ruled out VMware, the 2816, the laptop we used to test and ProviderA. So we figure it must 88.88.88.8. I have the old /27 that we are migrating from and I ask ProviderA to move it from the HQ circuit to the ColoCircuit and they do. I assign an IP address that has never been used to the server and I get, wait for it, the same error, same MAC address. I then assign another IP from the block, same error, same MAC, another IP and I get the same error, the same MAC. I tried 6 IPs and they all gave me the same result. Oh and before I moved that IP Block, I physically and personally removed it from all the servers it did reside on and flushed all ARP caches in my network and waited 30 minutes. (That was 4 hours ago and I just tested it and still the same results.)
Whiskey Tango Foxtrot...
That MAC address appears to be a Cisco Systems box (unless it's being spoofed...). I can't see any Cisco gear listed in your internal network configuration, so is it possible that your external provider has a piece of kit that has mistakenly claimed that IP instead of passing it through?
Agree with BlueCompute, that is a Cisco MAC. Could be that somebody has this box configured for proxy arp.
ASKER
When we plug a laptop directly into the ethernet hand-off from the provider the issue goes away so it can not be the Provider.
It's something on my network. We do not have any Cisco gear in our network. At all. And there is no way it is something that can be claiming one IP as a mistake, since we've tried 7 different IPs all with the same error reporting the same MAC.
So far the only thing I haven't tried is removing everything from the 2816 except the laptop and the ethernet hand-off, setting the 2816 up as a managed device, removing the point to point equipment off. I will doing all those tonight after hours since our network is now split between the two locations with this effort stalled out somewhere in the middle of it.
It's something on my network. We do not have any Cisco gear in our network. At all. And there is no way it is something that can be claiming one IP as a mistake, since we've tried 7 different IPs all with the same error reporting the same MAC.
So far the only thing I haven't tried is removing everything from the 2816 except the laptop and the ethernet hand-off, setting the 2816 up as a managed device, removing the point to point equipment off. I will doing all those tonight after hours since our network is now split between the two locations with this effort stalled out somewhere in the middle of it.
OK.
Can you confirm that you never see this MAC in any ARP tables?
That's a real bugger. I guess if you're down to physically unplugging switches now you could split the switch stack and test either half? Then half of them etc to narrow it down :(
Can you confirm that you never see this MAC in any ARP tables?
That's a real bugger. I guess if you're down to physically unplugging switches now you could split the switch stack and test either half? Then half of them etc to narrow it down :(
ASKER
Correct, never ever seen that MAC before and it doesn't match the only Cisco equipment we have, an old UCM server.
Yes Blue, I'll have to split stacks if it ends up pointing at that. First thing tonight, I'm going to just plug in my unmanaged switch, ethernet hand-off and a laptop. If I get error then we move to a managed 2816 and hope for the best. If that works then we plug in ethernet cables one at a time till I get a conflict and then we've found a new direction to research.
Yes Blue, I'll have to split stacks if it ends up pointing at that. First thing tonight, I'm going to just plug in my unmanaged switch, ethernet hand-off and a laptop. If I get error then we move to a managed 2816 and hope for the best. If that works then we plug in ethernet cables one at a time till I get a conflict and then we've found a new direction to research.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I am accepting my own response because I narrowed it down with other's help to the only conclusion that worked.