Wierd IP address Anamoly

I had a server with a static IP address, not a part of the DHCP scope.  We were having some issues with one of the applications on the server and the owner of the app suggested that the server be reformatted and reinstalled.  After reformatting the server and giving it a new IP address, someone decided to ping the old IP address of the now reformatted server and the IP pinged successfully to the device name the same as the server.  

They immediately said the server issue was due to an ip address conflict.  On the surface that is what it looks like, but here is the clincher, there is no device connected to this IP anywhere.  

To prove my point, I put the problem IP address into my laptop as a static IP and when I did an ipconfig, it showed up as my IP address.  When I had a co-worker ping the IP with the -a the device name was my computer.  My computer moved right along with this problem IP.  I changed my computer back to DHCP and picked up a new IP address.  Had my co-worker do another ping to the problem IP and it pinged with my device name.  

I am thinking there is something weird going on in my router or arp tables.  I need to get this resolved for my sanity sake.  Some powers at be want to blame this whole thing on an IP address, when I know it was not.
SalongeAsked:
Who is Participating?
 
giltjrCommented:
If it were not for the fact that IP address should not exist at all, showing up on multiple switches on different ports would be normal.

Say you had:

HOST1 <--> SW1 <--> SW2 <--> SW3 <--> HOST2

If you did a ping of host2 from host1, and they were in the same IP subnet.  Then:

On SW1 HOST2's address would show up on the port that connects SW1 and SW2.
On SW2 HOST2's address would show up on the port that connects SW2 and SW3.
On SW2 HOST2's address would show up on the port that HOST2 is on..

Of course in your case the IP address is not supposed to exist.  One of those ports should have a "computer" connected to it, while the other port should be the connection between the switches.

All you need to do now is figure out which port has a computer on it.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
My first thought was ARP table.
Does the server have the same name?
Could also check your DNS Forward Lookup Zone for the redundant server name with different IP's.
0
 
SalongeAuthor Commented:
The server was name the same, yes.  I checked the DNS Forward Lookup Zone and no there  no redundant servers with the same name and with different IPs.  The weird part is that I put the ip into my computer and everything worked fine.  Took it out and it still pings as my computer.  

I looked in the DNS server and the forward lookup zones and see several devices with the same IP address.
0
Network Scalability - Handle Complex Environments

Monitor your entire network from a single platform. Free 30 Day Trial Now!

 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Are any of those IP's being used by the former or rebuilt server?
0
 
SalongeAuthor Commented:
No.
0
 
SalongeAuthor Commented:
No, but the rebuilt server is using a new IP and I see in that Forward lookup I see another computer with the same IP.  I deleted the HOST (A) record for that.
0
 
AkinsdNetwork AdministratorCommented:
Do you have IP scanner Utility configured on any of your computers? If so, close out the utility and reboot that computer.

Also, try ipconfig /flushdns on a computer and try pinging the old address again.
0
 
SalongeAuthor Commented:
I shut down the IP scanner on my computer and rebooted my computer.  I did the ipconfig flushdns.  When I pinged the strange IP it pinged successfully naming my computer.
0
 
SalongeAuthor Commented:
My computer has a totally different IP address.  Why won't this IP go away?  It is connected to nothing.
0
 
AkinsdNetwork AdministratorCommented:
Interesting.
The last thing I can think of is to find a way to reboot your switches. I hope there is no Man-In-The-Middle or Spoofing on your network.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Is your computers' NIC single or dual port?
Does the Server have a multi-port NIC?
0
 
SalongeAuthor Commented:
How could I find out if anyone is spoofing on our network?
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Can you run a UNC path to that IP  \\xxx.xxx.xxx.xxx\c$ (administrative share)?
0
 
SalongeAuthor Commented:
Yes, there are dual nic cards on the server, but my laptop only has one.
0
 
SalongeAuthor Commented:
How would I run a UNC path?
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Are the dual NIC's on server separate or teamed?
Load balanced or redundant on fail? I would have thought that reset when the server was rebuilt.

Is there a secondary DNS that might have the server's IP assigned to another device.
Are you running a Reverse Lookup Zone?
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Click Start then Run for Windows XP or in Win7 just click Start and type:
 \\enter the IP here\c$

c$ is the administrative share. If a window opens then you should be able to determine which computer it is by user profiles, etc. If it's not associated with a computer then you'll see the message network path not found.
0
 
SalongeAuthor Commented:
When I do the UNC for the IP, I get nothing.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Was really just double checking that yours or any other computer was associated with that IP.

Have you tried pinging the server by name? What IP responds
Have you tried pinging your computer from the server by name or IP?
Did you check to see if you were using a DNS Reverse Lookup Zone.
0
 
SalongeAuthor Commented:
Have you tried pinging the server by name? What IP responds
Pinged the server by name and the correct new IP responds

Have you tried pinging your computer from the server by name or IP?
Pinged my computer by name and the correct IP responds

Did you check to see if you were using a DNS Reverse Lookup Zone.
I looked in our DNS Reverse Lookup Zone and the bogus IP PTR is my computer.
0
 
SalongeAuthor Commented:
I deleted the ptr record for the ip address, but I can still ping it.  It just doesn't have anything associated to it.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Narrowing it down... Is your switch/router a Cisco Catalyst?
0
 
SalongeAuthor Commented:
Yes, it is.  Also I ran a scan on the IP and my computer still showed up.  I clicked on my name and told the system to reboot the computer attached to that IP and my computer rebooted.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
Catalyst 6000/6500 series? Any VLAN's?
0
 
SalongeAuthor Commented:
No VLANS, 3560s
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
OK, I was reading a fixit how-to for the 6000/6500 series ARP-CAM problems.
I'm about the leave for the day...
I was about to ask if you multiple network configurations on the laptop then it just struck me... do you have wireless turned on your latop as well as the NIC?
0
 
AkinsdNetwork AdministratorCommented:
Reset the NIC on your computer.

Open command prompt as administrator
and issue the following commands
netsh int ip reset
netsh winsock reset
Shutdown your computer and try pinging the ip
Turn your computer on and ping again
0
 
SalongeAuthor Commented:
Ok I will try that in the morning and let you know
0
 
giltjrCommented:
If all your switches are managed, then ping the IP address.  Find the MAC and start looking at the mac address table until you find out the "last port" that MAC shows on.

Another possibility, is that somebody has configured a switch to do proxy arp and to respond to pings for that address.
0
 
SalongeAuthor Commented:
I performed the tasks of resetting my nic card and the phantom IP still thinks it is my IP address.  When I shut my computer down, I can still ping it.  When I ran an IP scan on that IP it says it is my computer with a wierd mac address.
0
 
SalongeAuthor Commented:
I looked at the ARP table in my router and the IP is showing up there and connected to that wierd mac address.
0
 
SalongeAuthor Commented:
I checked the Mac address in the mac address table on my main switch and it does not show up anywhere.
0
 
giltjrCommented:
Can you post the MAC address?
0
 
SalongeAuthor Commented:
00:22:64:1A:34:6C
0
 
giltjrCommented:
Do you have any HP equipment in your network?  That is a MAC address assigned to HP.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
00:22:64 is a known HP MAC address
0
 
SalongeAuthor Commented:
We have several HP's in our environment but none with that address.  My mac table does not find this address.  And since this IP thinks it is my computer, this is not my computer's mac address.
0
 
SalongeAuthor Commented:
how can I clear this ip our of my ARP table on my router to see if it finds it again?
0
 
giltjrCommented:
What type of router do you have?
0
 
SalongeAuthor Commented:
Cisco 1900 Series
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
I'm curious when you run ipconfig /all on your laptop... how many different connections do you have? Look for the 00:22:64 MAC address under any of the following:
Ethernet Adapter Bluetooth Network Connection
Wireless LAN Adapter Wireless Network Connection
Ethernet Adapter Local Area Connection
Tunnel Adapter Pseudo-Interface
0
 
SalongeAuthor Commented:
No, I don't.  I have a HP laptop also.  The interesting concept is that put this IP into my computer to prove to some others that there was not an IP conflict on a major server.  This used to be a server IP address.  Even after we formatted the server, we could ping the IP with -a and the server name would show up.  So I put the IP in my laptop and showed them that my computer did not come up with an IP conflict, nor was it getting kicked off.
0
 
giltjrCommented:
clear arp cache

Should clear all arp entries.  I don't think there is a way to clear an indvidual entry.  They should automatically get cleared out after awhile (I think 4-5 minutes) after not being referenced.

Stupid question.  By chance could there be a device mis-configured so that it thinks that IP address is a IP network broadcast IP address?  Example, if you have your IP subnet as 10.10.10.0/24, the IP address 10.10.10.255 is a broadcast address and if you ping it, devices will respond.  They will NOT report a IP address conflict, because they don't for broadcast addresses.
0
 
SalongeAuthor Commented:
I don't want to clear all right now midday.  I don't know what affect that will have on anything.  

As far as a mis-configured device, I don't think so.
0
 
SalongeAuthor Commented:
Will rebooting the router, clear the arp cache?
0
 
giltjrCommented:
Yes, but that will cause more problems than just clearing the arp cache.

-->  And since this IP thinks it is my computer, this is not my computer's mac address.

I must have missed this earlier.  Why do you say that the IP address thinks it belongs to your computer?
0
 
SalongeAuthor Commented:
It thinks it belongs to my computer because for about 10 minutes, it was the static IP of my computer.  I did this because it was initially a server IP address.  We were having problems with the server and someone said it was an IP conflict.  I said it was not, but something else.  We took the server down, reformatted it, and pinged the IP and it pinged successfully, proving that it was another device somewhere, I still disagreed.  Well, I knew that this was a good IP, not in a DHCP scope and no one else has access to do this.  So to prove that the IP was a good IP, i put it in my computer and my computer worked just fine with no IP conflict.  I am just trying to find out why this IP is out here connected to a phantom Mac address.  I believe it is because of something in my router. I don't want this issue to come and haunt me later.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
It stills sounds like DNS cache (ipconfig /flushdns) or ARP entry in the switch/router.
Is there a secondary DNS server where you could check your Forward and Reverse Lookup Zones like we did yesterday on your primary where you had enties with the IP associated to your computer?
0
 
SalongeAuthor Commented:
No we only have one DNS server.  I saw the entry in the ARP on the router, but nothing in the mac address table.  It is a mystery.
0
 
giltjrCommented:
To make sure I following this.

The address is actually ping'able when you believe that no host has this IP address assigned to it.  Correct?
0
 
SalongeAuthor Commented:
yes and I can run a IP scan on that IP and find a Mac address.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
So when you go to Network & Sharing Center - LAN Connection - Properties -  IPv4 Properties - Use the following DNS Server Addresses... there is no entry for Alternate DNS Server?
0
 
SalongeAuthor Commented:
That is correct.
0
 
giltjrCommented:
If you can find a mac address that address has to exist someplace.

You mentioned you had a Cisco 1900 router.  What type of switches do you use?

Is the "mystery" IP address on the same subnet as your normal IP address?
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
I believe the switch is a Cisco Catalyst 3500 series. I'd read a fixit article yesterday on the Catalyst 6000/6500 series ARP-CAM table issues.
0
 
SalongeAuthor Commented:
Yes the mystery IP is on the same subnet as all my other IP's.  The mac address is a mac address that has nothing connected to it.  We have Cisco Catalyst Switches 3560.  This is the same Mac address that showed up when it thought it was still connected to the server.
0
 
SalongeAuthor Commented:
So what is the fix or where can I get my hand on the article?
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
This is specific to the 6000\6500 series but references a number of troubleshooting steps for ARP and CAM:
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/71079-arp-cam-tableissues.html
0
 
giltjrCommented:
So on the 3650 you do a:

     show arp | i "mystery ip address"

Then you do:

      show mac address table | i "mac associated with mystery ip address"

The MAC does not show?
0
 
giltjrCommented:
Oh, you may want to do a:

show run | i "mystery ip address"

to make sure somebody did not create a static arp entry for that IP address.
0
 
SalongeAuthor Commented:
So on the 3650 you do a:

     show arp | i "mystery ip address"

Then you do:

      show mac address table | i "mac associated with mystery ip address"

The MAC does not show?

This is correct.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
giltjr,
I'm I accurate in stating that dynamic ARP entries assigned automatically clear/refresh in up to 20 minute intervals (default to 300 seconds) as long as the IP and MAC  address respond to the ARP sync; and static ARP entries manually added are persistent?
0
 
SalongeAuthor Commented:
Oh, you may want to do a:

show run | i "mystery ip address"

to make sure somebody did not create a static arp entry for that IP address.

I just did this and no information comes back.
0
 
giltjrCommented:
@Lee Ingalls
Static ARP entries are persistent.  They should show up when you do the "show mac address-table" and they should be shown as static/persistent.

Dynamic arp entries should stay in the table as long as the switch sees traffic FROM it within the timeout.  If, after the timeout period, the switch has not seen traffic from that mac, it will (should) be removed.

@Salonge

Do you only have one switch in your environment?

I would say you may need to open a TAC case with Cisco.  Off hand I can't think of a situation where a router/switch would have a arp entry for an IP address that has a mac address associated with it, but not have a mac address entry in the mac address table.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
One of the issues listed in the Catalyst 6000/6500 series ARP-CAM problems is "two entries show for MAC address in the MAC address table"; depending on which version salonge's 3560's ios is -- may have similar issues.

salonge what version is your switches firmware/ios?
I agree that  Cisco TAC may be the way to go... firmware/ios update may resolve.
0
 
SalongeAuthor Commented:
I have three managed switches in this building.  I did a ping test and got into my switches to look for the mac address.  The mac address showed up on all three switches on different ports.

Something is not right.
0
 
Lee IngallsDirector of IT/TS, Quality and FinanceCommented:
What IOS releases are your Cisco switches at?
0
 
AkinsdNetwork AdministratorCommented:
IP addresses do get stuck in device cache occasionally and sometimes, that's why I recommended resetting your NIC with the netsh commands.

What's strange here is the IP responds to ping even when the device is shutdown. The closest explanation to this puzzle is a device impersonation either via Man-In-The-Middle scenario or spoofing scenario. A device is responding for your computer somehow.

Wireshark may help reveal flow of packets but narrowing the attack down may be a little challenging. It is easier to set up preventative measures than attempting to narrow it down. at least from my perspective.

http://web.securityinnovation.com/appsec-weekly/blog/bid/63269/How-to-Test-for-Man-in-the-Middle-Vulnerabilities


Have you reloaded (rebooted) your switches yet?

Try the following
- It is not likely that alternative address is configured but it doesn't hurt to check either (on your PC and on the server)

- Perform a tracert from computer and/or traceroute from a switch to that IP

- Check if keep alive is disabled on your switch ports

- Find a good time to reboot all your devices.

- Perform updates where possible
0
 
SalongeAuthor Commented:
Thanks for everyone's help.  I wish I could award everyone the points because you all were so helpful.
0
 
giltjrCommented:
You can't award everybody 500 points, but you can split the points between multiple people.

You accept one as a answer, award that a portion of the 500 points, then accept others as assisted and award them portion of the 500 points.
0
 
SalongeAuthor Commented:
Can I still do that if I have closed the issue?
0
 
giltjrCommented:
You can make a request to have the question re-opened and then do it.

Go back up to the top of the page, and just below the box that has your question there is a yellow triangle "Request Attention", click on that and explain what you want to do.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.