?
Solved

Duplicate IP Detected on any machine, for any public ip, virtual or not

Posted on 2013-01-16
7
Medium Priority
?
810 Views
Last Modified: 2016-11-23
O.k. this is a good one.  Pull up an office chair and slam down a Mountain Dew, you are going to want to focus.

We have 2 datacenters, HQ & Colo.   Details below:

HQ:
30MB Internet from ProviderB (55.55.55.5/30 running BGP with a class C (99.99.99.0/24) pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.2), connected with an Adva (?) device on both ends.
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.101.0/24
10.1.102.0/24
10.1.103.0/24
10.1.104.0/24
These go into a big stack of Dell Powerconnects and a large VMWare Vcenter 5 farm.

(Migrating away from a 30MB non-BGP connection (22.22.22.2/27) that passes a /27 to me in bridged mode for publicly IPed servers which come straight from the 2816 to another unmanaged 2816 that then runs a network connection a virtual nic on each ESX host)

Colo:
30MB Internet from ProviderA (66.66.66.6/30 running BGP with a class C pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.3)
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.111.0/24
10.1.112.0/24
10.1.113.0/24
10.1.114.0/24
These go into a stack of Dell Powerconnects and a small but important VMWare Vcenter 5 farm.

The point to points are set up to route anything from the opposing network to the point to point gateway on the other side and all routes have been configured.  

Everything up to now runs like a dream, routing works on both sides, everything talks fine.  

We then went to move MS Lync from the HQ and move it to the Colo by setting up a NAT of 99.99.99.10 to 10.1.112.10 and it fails. (We could cover Lync and NAT for days and we would get nowhere, that problem is for another day.)  Long story short the only option is you can't NAT to a Lync Server that is handling incoming SIP requests so we call ProviderA and ask for 1 IP.  They agree and we get 88.88.88.8 assigned to us.  We assign it to the virtual Lync 2013 server and immediately get a Duplicate IP address error.  We check the windows logs and it says it's due to the MAC 64-D8-14-20-36-C2 has it.  That is weird since we've never used 88.88.88.8.  We checked all ARP tables on ALL switches, firewalls, VMs, ESXi hosts and can't find it at all.  We tried a brand new server that had just been built and we got the same error, same MAC.  We plug a laptop into the unmanaged 2816 and try that IP on it, Same error, same MAC.  We swap out the 2816 for a BRAND NEW 2816 plug the laptop in and get the same error, same MAC.

We plug the same laptop into the ethernet hand-off ProviderA gives us and it works like a charm.  No issues.  

So we ruled out VMware, the 2816, the laptop we used to test and ProviderA.  So we figure it must 88.88.88.8.  I have the old /27 that we are migrating from and I ask ProviderA to move it from the HQ circuit to the ColoCircuit and they do.  I assign an IP address that has never been used to the server and I get, wait for it, the same error, same MAC address.  I then assign another IP from the block, same error, same MAC, another IP and I get the same error, the same MAC.  I tried 6 IPs and they all gave me the same result.  Oh and before I moved that IP Block, I physically and personally removed it from all the servers it did reside on and flushed all ARP caches in my network and waited 30 minutes.  (That was 4 hours ago and I just tested it and still the same results.)

Whiskey Tango Foxtrot...
0
Comment
Question by:WorkSoft
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
7 Comments
 
LVL 14

Expert Comment

by:BlueCompute
ID: 38786775
That MAC address appears to be a Cisco Systems box (unless it's being spoofed...).  I can't see any Cisco gear listed in your internal network configuration, so is it possible that your external provider has a piece of kit that has mistakenly claimed that IP instead of passing it through?
0
 
LVL 57

Expert Comment

by:giltjr
ID: 38786823
Agree with BlueCompute, that is a Cisco MAC.  Could be that somebody has this box configured for proxy arp.
0
 

Author Comment

by:WorkSoft
ID: 38788121
When we plug a laptop directly into the ethernet hand-off from the provider the issue goes away so it can not be the Provider.

It's something on my network.  We do not have any Cisco gear in our network.  At all.  And there is no way it is something that can be claiming one IP as a mistake, since we've tried 7 different IPs all with the same error reporting the same MAC.

So far the only thing I haven't tried is removing everything from the 2816 except the laptop and the ethernet hand-off, setting the 2816 up as a managed device, removing the point to point equipment off.  I will doing all those tonight after hours since our network is now split between the two locations with this effort stalled out somewhere in the middle of it.
0
7 Extremely Useful Linux Commands for Beginners

Just getting started with Linux? Here's a quick start guide that has 7 commands that we believe will come in handy.

 
LVL 14

Expert Comment

by:BlueCompute
ID: 38788551
OK.

Can you confirm that you never see this MAC in any ARP tables?

That's a real bugger.  I guess if you're down to physically unplugging switches now you could split the switch stack and test either half?  Then half of them etc to narrow it down  :(
0
 

Author Comment

by:WorkSoft
ID: 38788654
Correct, never ever seen that MAC before and it doesn't match the only Cisco equipment we have, an old UCM server.

Yes Blue, I'll have to split stacks if it ends up pointing at that.  First thing tonight, I'm going to just plug in my unmanaged switch, ethernet hand-off and a laptop.  If I get error then we move to a managed 2816 and hope for the best.  If that works then we plug in ethernet cables one at a time till I get a conflict and then we've found a new direction to research.
0
 

Accepted Solution

by:
WorkSoft earned 0 total points
ID: 38857549
After multiple rounds of troubleshooting with multiple engineers, architects and support personnel, we found what we believe to be the issue.

What we believe happened was when a machine was IPed it tried to check the gateway and since the Adva device was connected to the same unmanaged switch as the gateway routers we believe the connection was sent across the Point to Point and back and the slight delay in response forced the machine to see it's self and thus pop the Dupe IP error.  Why we got that same MAC address is a mystery.

We took another unmanaged switch and put the ethernet/internet hand-off on one and the Point to Point on another and everything is now fine.  It makes no sense however, it's working and we have to move on.  Thanks to those that helped, I'll have a shot for you tonight!
0
 

Author Closing Comment

by:WorkSoft
ID: 38872820
I am accepting my own response because I narrowed it down with other's help to the only conclusion that worked.
0

Featured Post

Supports up to 4K resolution!

The VS192 2-Port 4K DisplayPort Splitter is perfect for anyone who needs to send one source of DisplayPort high definition video to two or four DisplayPort displays. The VS192 can split and also expand DisplayPort audio/video signal on two or four DisplayPort monitors.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you try to share a printer , you may receive one of the following error messages. Error message when you use the Add Printer Wizard to share a printer: Windows could not share your printer. Operation could not be completed (Error 0x000006…
Sometimes clients can lose connectivity with the Lotus Notes Domino Server, but there's not always an obvious answer as to why it happens.   Read this article to follow one of the first experiences I had with Lotus Notes on a client's machine, my…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…
Suggested Courses

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question