Solved

Duplicate IP Detected on any machine, for any public ip, virtual or not

Posted on 2013-01-16
7
782 Views
Last Modified: 2016-11-23
O.k. this is a good one.  Pull up an office chair and slam down a Mountain Dew, you are going to want to focus.

We have 2 datacenters, HQ & Colo.   Details below:

HQ:
30MB Internet from ProviderB (55.55.55.5/30 running BGP with a class C (99.99.99.0/24) pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.2), connected with an Adva (?) device on both ends.
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.101.0/24
10.1.102.0/24
10.1.103.0/24
10.1.104.0/24
These go into a big stack of Dell Powerconnects and a large VMWare Vcenter 5 farm.

(Migrating away from a 30MB non-BGP connection (22.22.22.2/27) that passes a /27 to me in bridged mode for publicly IPed servers which come straight from the 2816 to another unmanaged 2816 that then runs a network connection a virtual nic on each ESX host)

Colo:
30MB Internet from ProviderA (66.66.66.6/30 running BGP with a class C pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.3)
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.111.0/24
10.1.112.0/24
10.1.113.0/24
10.1.114.0/24
These go into a stack of Dell Powerconnects and a small but important VMWare Vcenter 5 farm.

The point to points are set up to route anything from the opposing network to the point to point gateway on the other side and all routes have been configured.  

Everything up to now runs like a dream, routing works on both sides, everything talks fine.  

We then went to move MS Lync from the HQ and move it to the Colo by setting up a NAT of 99.99.99.10 to 10.1.112.10 and it fails. (We could cover Lync and NAT for days and we would get nowhere, that problem is for another day.)  Long story short the only option is you can't NAT to a Lync Server that is handling incoming SIP requests so we call ProviderA and ask for 1 IP.  They agree and we get 88.88.88.8 assigned to us.  We assign it to the virtual Lync 2013 server and immediately get a Duplicate IP address error.  We check the windows logs and it says it's due to the MAC 64-D8-14-20-36-C2 has it.  That is weird since we've never used 88.88.88.8.  We checked all ARP tables on ALL switches, firewalls, VMs, ESXi hosts and can't find it at all.  We tried a brand new server that had just been built and we got the same error, same MAC.  We plug a laptop into the unmanaged 2816 and try that IP on it, Same error, same MAC.  We swap out the 2816 for a BRAND NEW 2816 plug the laptop in and get the same error, same MAC.

We plug the same laptop into the ethernet hand-off ProviderA gives us and it works like a charm.  No issues.  

So we ruled out VMware, the 2816, the laptop we used to test and ProviderA.  So we figure it must 88.88.88.8.  I have the old /27 that we are migrating from and I ask ProviderA to move it from the HQ circuit to the ColoCircuit and they do.  I assign an IP address that has never been used to the server and I get, wait for it, the same error, same MAC address.  I then assign another IP from the block, same error, same MAC, another IP and I get the same error, the same MAC.  I tried 6 IPs and they all gave me the same result.  Oh and before I moved that IP Block, I physically and personally removed it from all the servers it did reside on and flushed all ARP caches in my network and waited 30 minutes.  (That was 4 hours ago and I just tested it and still the same results.)

Whiskey Tango Foxtrot...
0
Comment
Question by:WorkSoft
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
7 Comments
 
LVL 14

Expert Comment

by:BlueCompute
ID: 38786775
That MAC address appears to be a Cisco Systems box (unless it's being spoofed...).  I can't see any Cisco gear listed in your internal network configuration, so is it possible that your external provider has a piece of kit that has mistakenly claimed that IP instead of passing it through?
0
 
LVL 57

Expert Comment

by:giltjr
ID: 38786823
Agree with BlueCompute, that is a Cisco MAC.  Could be that somebody has this box configured for proxy arp.
0
 

Author Comment

by:WorkSoft
ID: 38788121
When we plug a laptop directly into the ethernet hand-off from the provider the issue goes away so it can not be the Provider.

It's something on my network.  We do not have any Cisco gear in our network.  At all.  And there is no way it is something that can be claiming one IP as a mistake, since we've tried 7 different IPs all with the same error reporting the same MAC.

So far the only thing I haven't tried is removing everything from the 2816 except the laptop and the ethernet hand-off, setting the 2816 up as a managed device, removing the point to point equipment off.  I will doing all those tonight after hours since our network is now split between the two locations with this effort stalled out somewhere in the middle of it.
0
Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

 
LVL 14

Expert Comment

by:BlueCompute
ID: 38788551
OK.

Can you confirm that you never see this MAC in any ARP tables?

That's a real bugger.  I guess if you're down to physically unplugging switches now you could split the switch stack and test either half?  Then half of them etc to narrow it down  :(
0
 

Author Comment

by:WorkSoft
ID: 38788654
Correct, never ever seen that MAC before and it doesn't match the only Cisco equipment we have, an old UCM server.

Yes Blue, I'll have to split stacks if it ends up pointing at that.  First thing tonight, I'm going to just plug in my unmanaged switch, ethernet hand-off and a laptop.  If I get error then we move to a managed 2816 and hope for the best.  If that works then we plug in ethernet cables one at a time till I get a conflict and then we've found a new direction to research.
0
 

Accepted Solution

by:
WorkSoft earned 0 total points
ID: 38857549
After multiple rounds of troubleshooting with multiple engineers, architects and support personnel, we found what we believe to be the issue.

What we believe happened was when a machine was IPed it tried to check the gateway and since the Adva device was connected to the same unmanaged switch as the gateway routers we believe the connection was sent across the Point to Point and back and the slight delay in response forced the machine to see it's self and thus pop the Dupe IP error.  Why we got that same MAC address is a mystery.

We took another unmanaged switch and put the ethernet/internet hand-off on one and the Point to Point on another and everything is now fine.  It makes no sense however, it's working and we have to move on.  Thanks to those that helped, I'll have a shot for you tonight!
0
 

Author Closing Comment

by:WorkSoft
ID: 38872820
I am accepting my own response because I narrowed it down with other's help to the only conclusion that worked.
0

Featured Post

Create the perfect environment for any meeting

You might have a modern environment with all sorts of high-tech equipment, but what makes it worthwhile is how you seamlessly bring together the presentation with audio, video and lighting. The ATEN Control System provides integrated control and system automation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Ping and real time 48 79
Is Fedora an appropriate distro for the environment. 7 88
Sonos and 5ghz 14 43
Exchange 2016 - not receiving mail 17 37
ADCs have gained traction within the last decade, largely due to increased demand for legacy load balancing appliances to handle more advanced application delivery requirements and improve application performance.
I had an issue with InstallShield not being able to use Computer Browser service on Windows Server 2012. Here is the solution I found.
Viewers will learn how to connect to a wireless network using the network security key. They will also learn how to access the IP address and DNS server for connections that must be done manually. After setting up a router, find the network security…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

735 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question