Duplicate IP Detected on any machine, for any public ip, virtual or not

O.k. this is a good one.  Pull up an office chair and slam down a Mountain Dew, you are going to want to focus.

We have 2 datacenters, HQ & Colo.   Details below:

30MB Internet from ProviderB ( running BGP with a class C ( pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (, connected with an Adva (?) device on both ends.
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
These go into a big stack of Dell Powerconnects and a large VMWare Vcenter 5 farm.

(Migrating away from a 30MB non-BGP connection ( that passes a /27 to me in bridged mode for publicly IPed servers which come straight from the 2816 to another unmanaged 2816 that then runs a network connection a virtual nic on each ESX host)

30MB Internet from ProviderA ( running BGP with a class C pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
These go into a stack of Dell Powerconnects and a small but important VMWare Vcenter 5 farm.

The point to points are set up to route anything from the opposing network to the point to point gateway on the other side and all routes have been configured.  

Everything up to now runs like a dream, routing works on both sides, everything talks fine.  

We then went to move MS Lync from the HQ and move it to the Colo by setting up a NAT of to and it fails. (We could cover Lync and NAT for days and we would get nowhere, that problem is for another day.)  Long story short the only option is you can't NAT to a Lync Server that is handling incoming SIP requests so we call ProviderA and ask for 1 IP.  They agree and we get assigned to us.  We assign it to the virtual Lync 2013 server and immediately get a Duplicate IP address error.  We check the windows logs and it says it's due to the MAC 64-D8-14-20-36-C2 has it.  That is weird since we've never used  We checked all ARP tables on ALL switches, firewalls, VMs, ESXi hosts and can't find it at all.  We tried a brand new server that had just been built and we got the same error, same MAC.  We plug a laptop into the unmanaged 2816 and try that IP on it, Same error, same MAC.  We swap out the 2816 for a BRAND NEW 2816 plug the laptop in and get the same error, same MAC.

We plug the same laptop into the ethernet hand-off ProviderA gives us and it works like a charm.  No issues.  

So we ruled out VMware, the 2816, the laptop we used to test and ProviderA.  So we figure it must  I have the old /27 that we are migrating from and I ask ProviderA to move it from the HQ circuit to the ColoCircuit and they do.  I assign an IP address that has never been used to the server and I get, wait for it, the same error, same MAC address.  I then assign another IP from the block, same error, same MAC, another IP and I get the same error, the same MAC.  I tried 6 IPs and they all gave me the same result.  Oh and before I moved that IP Block, I physically and personally removed it from all the servers it did reside on and flushed all ARP caches in my network and waited 30 minutes.  (That was 4 hours ago and I just tested it and still the same results.)

Whiskey Tango Foxtrot...
Who is Participating?
WorkSoftConnect With a Mentor Author Commented:
After multiple rounds of troubleshooting with multiple engineers, architects and support personnel, we found what we believe to be the issue.

What we believe happened was when a machine was IPed it tried to check the gateway and since the Adva device was connected to the same unmanaged switch as the gateway routers we believe the connection was sent across the Point to Point and back and the slight delay in response forced the machine to see it's self and thus pop the Dupe IP error.  Why we got that same MAC address is a mystery.

We took another unmanaged switch and put the ethernet/internet hand-off on one and the Point to Point on another and everything is now fine.  It makes no sense however, it's working and we have to move on.  Thanks to those that helped, I'll have a shot for you tonight!
That MAC address appears to be a Cisco Systems box (unless it's being spoofed...).  I can't see any Cisco gear listed in your internal network configuration, so is it possible that your external provider has a piece of kit that has mistakenly claimed that IP instead of passing it through?
Agree with BlueCompute, that is a Cisco MAC.  Could be that somebody has this box configured for proxy arp.
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

WorkSoftAuthor Commented:
When we plug a laptop directly into the ethernet hand-off from the provider the issue goes away so it can not be the Provider.

It's something on my network.  We do not have any Cisco gear in our network.  At all.  And there is no way it is something that can be claiming one IP as a mistake, since we've tried 7 different IPs all with the same error reporting the same MAC.

So far the only thing I haven't tried is removing everything from the 2816 except the laptop and the ethernet hand-off, setting the 2816 up as a managed device, removing the point to point equipment off.  I will doing all those tonight after hours since our network is now split between the two locations with this effort stalled out somewhere in the middle of it.

Can you confirm that you never see this MAC in any ARP tables?

That's a real bugger.  I guess if you're down to physically unplugging switches now you could split the switch stack and test either half?  Then half of them etc to narrow it down  :(
WorkSoftAuthor Commented:
Correct, never ever seen that MAC before and it doesn't match the only Cisco equipment we have, an old UCM server.

Yes Blue, I'll have to split stacks if it ends up pointing at that.  First thing tonight, I'm going to just plug in my unmanaged switch, ethernet hand-off and a laptop.  If I get error then we move to a managed 2816 and hope for the best.  If that works then we plug in ethernet cables one at a time till I get a conflict and then we've found a new direction to research.
WorkSoftAuthor Commented:
I am accepting my own response because I narrowed it down with other's help to the only conclusion that worked.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.