Solved

Duplicate IP Detected on any machine, for any public ip, virtual or not

Posted on 2013-01-16
7
744 Views
Last Modified: 2016-11-23
O.k. this is a good one.  Pull up an office chair and slam down a Mountain Dew, you are going to want to focus.

We have 2 datacenters, HQ & Colo.   Details below:

HQ:
30MB Internet from ProviderB (55.55.55.5/30 running BGP with a class C (99.99.99.0/24) pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.2), connected with an Adva (?) device on both ends.
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.101.0/24
10.1.102.0/24
10.1.103.0/24
10.1.104.0/24
These go into a big stack of Dell Powerconnects and a large VMWare Vcenter 5 farm.

(Migrating away from a 30MB non-BGP connection (22.22.22.2/27) that passes a /27 to me in bridged mode for publicly IPed servers which come straight from the 2816 to another unmanaged 2816 that then runs a network connection a virtual nic on each ESX host)

Colo:
30MB Internet from ProviderA (66.66.66.6/30 running BGP with a class C pointed at the single ip provided by the /30) & a 100mb Point to Point E-Line (straight ethernet hand-off) (10.1.100.3)
Both connections plug into an unmanaged Dell Powerconnect 2816 then out to a pair of Watchgaurd 535s in Active/Passive mode.
LAN is
10.1.111.0/24
10.1.112.0/24
10.1.113.0/24
10.1.114.0/24
These go into a stack of Dell Powerconnects and a small but important VMWare Vcenter 5 farm.

The point to points are set up to route anything from the opposing network to the point to point gateway on the other side and all routes have been configured.  

Everything up to now runs like a dream, routing works on both sides, everything talks fine.  

We then went to move MS Lync from the HQ and move it to the Colo by setting up a NAT of 99.99.99.10 to 10.1.112.10 and it fails. (We could cover Lync and NAT for days and we would get nowhere, that problem is for another day.)  Long story short the only option is you can't NAT to a Lync Server that is handling incoming SIP requests so we call ProviderA and ask for 1 IP.  They agree and we get 88.88.88.8 assigned to us.  We assign it to the virtual Lync 2013 server and immediately get a Duplicate IP address error.  We check the windows logs and it says it's due to the MAC 64-D8-14-20-36-C2 has it.  That is weird since we've never used 88.88.88.8.  We checked all ARP tables on ALL switches, firewalls, VMs, ESXi hosts and can't find it at all.  We tried a brand new server that had just been built and we got the same error, same MAC.  We plug a laptop into the unmanaged 2816 and try that IP on it, Same error, same MAC.  We swap out the 2816 for a BRAND NEW 2816 plug the laptop in and get the same error, same MAC.

We plug the same laptop into the ethernet hand-off ProviderA gives us and it works like a charm.  No issues.  

So we ruled out VMware, the 2816, the laptop we used to test and ProviderA.  So we figure it must 88.88.88.8.  I have the old /27 that we are migrating from and I ask ProviderA to move it from the HQ circuit to the ColoCircuit and they do.  I assign an IP address that has never been used to the server and I get, wait for it, the same error, same MAC address.  I then assign another IP from the block, same error, same MAC, another IP and I get the same error, the same MAC.  I tried 6 IPs and they all gave me the same result.  Oh and before I moved that IP Block, I physically and personally removed it from all the servers it did reside on and flushed all ARP caches in my network and waited 30 minutes.  (That was 4 hours ago and I just tested it and still the same results.)

Whiskey Tango Foxtrot...
0
Comment
Question by:WorkSoft
  • 4
  • 2
7 Comments
 
LVL 14

Expert Comment

by:BlueCompute
Comment Utility
That MAC address appears to be a Cisco Systems box (unless it's being spoofed...).  I can't see any Cisco gear listed in your internal network configuration, so is it possible that your external provider has a piece of kit that has mistakenly claimed that IP instead of passing it through?
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Agree with BlueCompute, that is a Cisco MAC.  Could be that somebody has this box configured for proxy arp.
0
 

Author Comment

by:WorkSoft
Comment Utility
When we plug a laptop directly into the ethernet hand-off from the provider the issue goes away so it can not be the Provider.

It's something on my network.  We do not have any Cisco gear in our network.  At all.  And there is no way it is something that can be claiming one IP as a mistake, since we've tried 7 different IPs all with the same error reporting the same MAC.

So far the only thing I haven't tried is removing everything from the 2816 except the laptop and the ethernet hand-off, setting the 2816 up as a managed device, removing the point to point equipment off.  I will doing all those tonight after hours since our network is now split between the two locations with this effort stalled out somewhere in the middle of it.
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 14

Expert Comment

by:BlueCompute
Comment Utility
OK.

Can you confirm that you never see this MAC in any ARP tables?

That's a real bugger.  I guess if you're down to physically unplugging switches now you could split the switch stack and test either half?  Then half of them etc to narrow it down  :(
0
 

Author Comment

by:WorkSoft
Comment Utility
Correct, never ever seen that MAC before and it doesn't match the only Cisco equipment we have, an old UCM server.

Yes Blue, I'll have to split stacks if it ends up pointing at that.  First thing tonight, I'm going to just plug in my unmanaged switch, ethernet hand-off and a laptop.  If I get error then we move to a managed 2816 and hope for the best.  If that works then we plug in ethernet cables one at a time till I get a conflict and then we've found a new direction to research.
0
 

Accepted Solution

by:
WorkSoft earned 0 total points
Comment Utility
After multiple rounds of troubleshooting with multiple engineers, architects and support personnel, we found what we believe to be the issue.

What we believe happened was when a machine was IPed it tried to check the gateway and since the Adva device was connected to the same unmanaged switch as the gateway routers we believe the connection was sent across the Point to Point and back and the slight delay in response forced the machine to see it's self and thus pop the Dupe IP error.  Why we got that same MAC address is a mystery.

We took another unmanaged switch and put the ethernet/internet hand-off on one and the Point to Point on another and everything is now fine.  It makes no sense however, it's working and we have to move on.  Thanks to those that helped, I'll have a shot for you tonight!
0
 

Author Closing Comment

by:WorkSoft
Comment Utility
I am accepting my own response because I narrowed it down with other's help to the only conclusion that worked.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

What’s a web proxy server? A proxy server is a server that goes between clients and web servers, used in corporate to enforce corporate browsing policy and ensure security. Proxy servers are commonly used in three modes. A)    Forward proxy …
Creating an OSPF network that automatically (dynamically) reroutes network traffic over other connections to prevent network downtime.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now