Unable to locate device pulling same DHCP IP as one of our Meshed AP

Andrew N. Kowtalo
Andrew N. Kowtalo used Ask the Experts™
on
Our wireless vendor is currently telling us that the DHCP IP addressed assigned to one of our meshed Access Points is in conflict with another device on the same network with a specific MAC address.   We have looked in DHCP and DNS on our Domain controller and have been unable to locate this specific MAC linked to this IP From the ARP table they provided us.  

Can anyone tell me a command I can run that can show me the MAC and IP along with the device name the arp table reports on the wireless Access Point so I can try and delete it?   This is causing a severe conflict.

(192.10.10.82) at f4:ea:67:7c:b2:23 [ether] on eth0 <--- Not AP1
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Paul MacDonaldDirector, Information Systems

Commented:
There's no indication what OS you're on, but I'm guessing some variation of Linux.

The device name is moot if you have the IP address (and/or the MAC address).  ARP will give you IP and MAC (as you know), but does it show the duplicate entry?  

DHCP should tell you the IP and MAC for its clients. If you can't find the duplicate entry in DHCP, either the device is using a static IP or is getting its IP from another DHCP server.  I might create a reservation for the IP address in question (so DHCP can't hand it out), then wait and see if a device continues to use that IP address (which would tell you it was static).  Unless you have a ridiculous number of APs, I would be inclined to connect to them one-at-a-time, and then have them release and renew their DHCP address (maybe just power-cycle them?).
Andrew N. KowtaloSupport Center Engineer

Author

Commented:
I am performing this from a Windows DC to try and locate that specific mac.   Here is what the vendor is saying.

Hello Andrew and Michael,

I took a look over this case and would like to point out a few things to clear up some confusion that seems to be present.

Our APs do not have a NAT and should not have a NAT configured for them to communicate with us. They are meant to be on the network as a regular device like a workstation and just need the firewall rules in place so that they can utilize the Router's internet connection to communicate with us. Based on Michael's update from about an hour ago I just wanted to make sure that this was known and understood.

And then somewhat related to that, I do need to mention that we have seen some issues with our APs having IPs outside the standard internal IP Schemes such as 192.168.x.x. More specifically with SSIDs that get bridged to the LAN. Since most of these APs have a 192.10.x.x IP which are all registered Public IPs. You may not have this issue at the moment but I did need to mention that in case it comes up later on.

Going back to what Cheryl was mentioning with the IP conflict, currently in the network manager AP1 shows that the last known IP is 192.10.10.82. We were able to find that there is a Datto Backup Device that is on the same network as these Maple APs. And when we connect to that Backup Device and ping the 192.10.10.82 and then check the Arp Table immediately afterward we see that it is not the AP1 MAC but f4:ea:67:7c:b2:23 which signals that there is likely two devices that are requesting the same IP which would cause the IP conflict we mentioned. And since we have been seeing this after checking a few times it seems to be happening frequently. And checking the cloud connection graph for the AP, it is definitely missing quite a few checkins which reinforces this theory of an IP conflict.

Here is another recent pull of this data (screenshot attached to show Current LAN IPs)

PING 192.10.10.82 (192.10.10.82) 56(84) bytes of data.
64 bytes from 192.10.10.82: icmp_seq=1 ttl=64 time=0.908 ms
64 bytes from 192.10.10.82: icmp_seq=2 ttl=64 time=0.895 ms
64 bytes from 192.10.10.82: icmp_seq=3 ttl=64 time=0.949 ms
64 bytes from 192.10.10.82: icmp_seq=4 ttl=64 time=0.967 ms
64 bytes from 192.10.10.82: icmp_seq=5 ttl=64 time=0.932 ms
64 bytes from 192.10.10.82: icmp_seq=6 ttl=64 time=0.912 ms
^C
--- 192.10.10.82 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5060ms
rtt min/avg/max/mdev = 0.895/0.927/0.967/0.030 ms
? (192.10.10.62) at 80:e8:2c:11:20:c9 [ether] on eth0? (192.10.10.98) at 80:e8:2c:11:15:f8 [ether] on eth0
? (192.10.10.89) at f4:39:09:17:43:71 [ether] on eth0
? (192.10.10.15) at 00:0c:29:e5:91:c9 [ether] on eth0
? (192.10.10.60) at 80:e8:2c:11:23:5c [ether] on eth0
? (192.10.10.115) at c4:34:6b:b7:48:20 [ether] on eth0
? (192.10.10.96) at ac:86:74:fb:23:c0 [ether] on eth0 <--- AP3

? (192.10.10.68) at 80:e8:2c:11:19:11 [ether] on eth0
? (192.10.10.58) at 80:e8:2c:11:18:6e [ether] on eth0
? (192.10.10.66) at 80:e8:2c:11:18:42 [ether] on eth0
? (192.10.10.111) at 80:e8:2c:16:b9:71 [ether] on eth0
? (192.10.10.64) at 20:7d:74:47:4e:17 [ether] on eth0
? (192.10.10.109) at 80:e8:2c:11:19:18 [ether] on eth0
? (192.10.10.7) at 02:bf:c0:0a:0a:08 [ether] on eth0
? (192.10.10.5) at 00:90:7f:87:3d:3e [ether] on eth0
? (192.10.10.105) at 40:a8:f0:46:b3:b6 [ether] on eth0
? (192.10.10.77) at 00:60:b9:9d:d3:bd [ether] on eth0
? (192.10.10.3) at 78:e3:b5:e7:e6:3c [ether] on eth0
? (192.10.10.20) at 90:e2:ba:06:fd:60 [ether] on eth0
? (192.10.10.75) at 20:47:47:b3:ad:b1 [ether] on eth0
? (192.10.10.92) at c8:d3:ff:9f:1d:2e [ether] on eth0
? (192.10.10.73) at 80:e8:2c:11:18:8d [ether] on eth0
? (192.10.10.63) at 80:e8:2c:11:23:8c [ether] on eth0
? (192.10.10.16) at 00:15:5d:0b:1d:32 [ether] on eth0
? (192.10.10.71) at 24:a0:74:6d:73:6e [ether] on eth0
? (192.10.10.61) at 10:30:25:92:8d:61 [ether] on eth0
? (192.10.10.88) at b0:0c:d1:5e:f6:61 [ether] on eth0
? (192.10.10.14) at 3c:4a:92:e6:14:1a [ether] on eth0
? (192.10.10.59) at cc:d2:81:59:90:29 [ether] on eth0
? (192.10.10.12) at 3c:4a:92:e6:a2:88 [ether] on eth0
? (192.10.10.57) at 98:10:e8:e2:73:b5 [ether] on eth0
? (192.10.10.10) at fc:15:b4:0f:1e:d8 [ether] on eth0
? (192.10.10.82) at f4:ea:67:7c:b2:23 [ether] on eth0 <--- Not AP1

? (192.10.10.108) at 80:e8:2c:11:21:92 [ether] on eth0
? (192.10.10.80) at 2c:27:d7:26:5c:fd [ether] on eth0
? (192.10.10.6) at 00:23:7d:61:25:f9 [ether] on eth0
? (192.10.10.78) at c4:65:16:b0:69:f3 [ether] on eth0
? (192.10.10.4) at 02:bf:c0:0a:0a:08 [ether] on eth0
? (192.10.10.95) at ac:86:74:fb:24:00 [ether] on eth0 <--- AP2

? (192.10.10.104) at 80:e8:2c:11:14:d8 [ether] on eth0
? (192.10.10.21) at 00:15:5d:0b:1d:2f [ether] on eth0
? (192.10.10.76) at 80:e8:2c:11:23:52 [ether] on eth0
? (192.10.10.102) at 80:e8:2c:11:21:b9 [ether] on eth0
? (192.10.10.93) at f4:39:09:17:42:1d [ether] on eth0
? (192.10.10.91) at 9c:7b:ef:a8:3d:99 [ether] on eth0
? (192.10.10.72) at 80:e8:2c:11:1e:8a [ether] on eth0


I would also like to ask a bit more information about how this site specifically is set up since there obviously can't be two devices with the same 192.10.10.82 IP. As Cheryl noted it appears to be a Cisco device. Can you confirm this is not another router or layer 3 switch?

And is it safe to assume that the other locations in this network are set up in a similar fashion? I see that most of them are using a similar IP scheme but without the Firewall rules for the Advanced Troubleshooting, we cannot connect to the APs really do any kind of accurate troubleshooting. And taking a look in the same way we found the Maple BDR Device, we do not see a BDR device at the other sites so we have no form of accessing those devices.

And tying this all back to the issue about updating the APs to 6.5.3, there are clearly a few issues that exist currently that would need to get addressed before taking into consideration the updated firmware. The issues we see here currently could also be behind what happened when you updated.

Please let us know if you have any questions.

Thank you



This was all gathered from the AP ARP table
Andrew N. KowtaloSupport Center Engineer

Author

Commented:
Our Reply

These devices are on the customer network just like a work station, they receive DHCP and when they “talk” to the public world their communication is translated to the public IP address of the firewall just like any PC on the network. It is physically impossible for the public world to return traffic to these devices on their internal private IP address. I complete understand that the customers network utilizes none standard private IP addressing and I wish there was a way to change this but at this time we do not.
If I read your email proper are you stating there is another Datto customer with a backup device registering to the 192.10.10.82 address in your cloud environment? I understand that could be an issue if this AP actually sat on the public internet but it is in a private network masked behind a firewall. Since your service resides on the public internet and these devices initiate the traffic through the firewall the only address your cloud should care about is the public IP address these devices translate to when communicating outbound through the firewall correct, not the internal IP address? I feel like there is something I am missing here
Andrew N. KowtaloSupport Center Engineer

Author

Commented:
Vendor Reply

For the NAT part, I just wanted to make sure we were on the same page since it sounded like you either thought there was a preconfigured NAT or that you had a NAT configured for the APs to communicate through.

Then the Datto Device appears to be yours as it is registered with you, it is device  (002590E4E466) You should be able to see that in your BCDR Status page on the Partner Portal.  (Which we do see)

Then in terms of the IP of the actual AP, you are correct, that alone usually would not impact the cloud connection. The Issues I was mentioning were more so about the actual AP working within the network when you bridge the SSID to a LAN or VLAN. I just needed to mention that in case that issue comes up later one.

Going back to the more dominant issues for what we are currently looking at, the IP conflict and the Rules. I know it was mentioned that the Advanced Troubleshooting rules could not get put in place at this time. So we can put that on the back burner for now but for more precise troubleshooting with the sites outside the Maple location, we will need to revisit that and figure out something for this since we won't be able to connect to those APs.

For the IP Conflict specifically, can you confirm what device we are seeing when we ping the .82 address and get the f4:ea:67:7c:b2:23 MAC address?
Andrew N. KowtaloSupport Center Engineer

Author

Commented:
So bottom line there is a conflicting MAC address they are seeing on the AP ARP table that we cannot locate.  Ergo we can not reboot anything or assign the AP a different IP from a simple reboot because our DHCP server will reassign the same IP to that device if its not taken from another device.

Honestly I think there is an issue on their end.   But no way to prove it.
Paul MacDonaldDirector, Information Systems

Commented:
In the MAC addresses "ac:86:74" represents the manufacturer of your AP.  f4:ea:67 is a Cisco device so it's definitely not an Open Mesh AP.  

I expect you have a finite number of Cisco devices and it should be fairly easy to inspect them for their MACs.  Another alternative might be to create a reservation in DHCP for the problem IP address so it simply can't be assigned dynamically.  If there's a device with that IP assigned statically, it will keep using that IP address and the other device can get a new DHCP one.  Simply wait for the lease to expire or reboot your APs until the problem goes away.
Andrew N. KowtaloSupport Center Engineer

Author

Commented:
One response from the vendor said Additionally, the 192.10.10.82 address is a public address belonging to a company called SYMBOLICS1.
Paul MacDonaldDirector, Information Systems

Commented:
The vendor is largely right about your using 192.10.x.y.  You're fine as long as you're certain the range remains private, but it could cause issues if someone on your network needed to get to "Symbolics", and their computer couldn't find a proper route because you're using their address space.  Beyond that, I don't think it pertains to this particular issue.
Andrew N. KowtaloSupport Center Engineer

Author

Commented:
Thanks Paul.  I went ahead and created an exception range on the DC for 192.10.10.82 ---> 192.10.10.82 rebooted the AP and lets see what happens.
Top Expert 2014

Commented:
Paul's advice is good.

You could easily create a reservation for the MAC of the AP so it's assigned a different IP, or just add the IP to an exclusion range.

Have you checked the configuration on your Cisco devices to see if any are assigned the offending IP statically?  If you only have a few this is an easy place to start.  Perhaps someone connected a rogue device to your network?

Run the following command at various times to see if the associated MAC changes;
arp -a 192.10.10.82
If you have smart switches, you can use a procedure like below to track down the port a device is connected to.  Commands would likely vary depending on manufacturer.
https://kellerschroeder.zendesk.com/hc/en-us/articles/214975386-How-to-Track-Down-a-Switch-Port


Edit:  I see there was some progress on this thread since I started typing up this post, so some of the advice has already been covered...
Support Center Engineer
Commented:
added exception to dhcp
Paul MacDonaldDirector, Information Systems

Commented:
So you're crediting yourself with the solution?

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial