Solved

Wierd IP address Anamoly

Posted on 2014-02-27
73
342 Views
Last Modified: 2014-03-14
I had a server with a static IP address, not a part of the DHCP scope.  We were having some issues with one of the applications on the server and the owner of the app suggested that the server be reformatted and reinstalled.  After reformatting the server and giving it a new IP address, someone decided to ping the old IP address of the now reformatted server and the IP pinged successfully to the device name the same as the server.  

They immediately said the server issue was due to an ip address conflict.  On the surface that is what it looks like, but here is the clincher, there is no device connected to this IP anywhere.  

To prove my point, I put the problem IP address into my laptop as a static IP and when I did an ipconfig, it showed up as my IP address.  When I had a co-worker ping the IP with the -a the device name was my computer.  My computer moved right along with this problem IP.  I changed my computer back to DHCP and picked up a new IP address.  Had my co-worker do another ping to the problem IP and it pinged with my device name.  

I am thinking there is something weird going on in my router or arp tables.  I need to get this resolved for my sanity sake.  Some powers at be want to blame this whole thing on an IP address, when I know it was not.
0
Comment
Question by:Salonge
  • 35
  • 19
  • 14
  • +1
73 Comments
 
LVL 7

Assisted Solution

by:Lee Ingalls
Lee Ingalls earned 167 total points
Comment Utility
My first thought was ARP table.
Does the server have the same name?
Could also check your DNS Forward Lookup Zone for the redundant server name with different IP's.
0
 

Author Comment

by:Salonge
Comment Utility
The server was name the same, yes.  I checked the DNS Forward Lookup Zone and no there  no redundant servers with the same name and with different IPs.  The weird part is that I put the ip into my computer and everything worked fine.  Took it out and it still pings as my computer.  

I looked in the DNS server and the forward lookup zones and see several devices with the same IP address.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Are any of those IP's being used by the former or rebuilt server?
0
 

Author Comment

by:Salonge
Comment Utility
No.
0
 

Author Comment

by:Salonge
Comment Utility
No, but the rebuilt server is using a new IP and I see in that Forward lookup I see another computer with the same IP.  I deleted the HOST (A) record for that.
0
 
LVL 18

Assisted Solution

by:Akinsd
Akinsd earned 166 total points
Comment Utility
Do you have IP scanner Utility configured on any of your computers? If so, close out the utility and reboot that computer.

Also, try ipconfig /flushdns on a computer and try pinging the old address again.
0
 

Author Comment

by:Salonge
Comment Utility
I shut down the IP scanner on my computer and rebooted my computer.  I did the ipconfig flushdns.  When I pinged the strange IP it pinged successfully naming my computer.
0
 

Author Comment

by:Salonge
Comment Utility
My computer has a totally different IP address.  Why won't this IP go away?  It is connected to nothing.
0
 
LVL 18

Expert Comment

by:Akinsd
Comment Utility
Interesting.
The last thing I can think of is to find a way to reboot your switches. I hope there is no Man-In-The-Middle or Spoofing on your network.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Is your computers' NIC single or dual port?
Does the Server have a multi-port NIC?
0
 

Author Comment

by:Salonge
Comment Utility
How could I find out if anyone is spoofing on our network?
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Can you run a UNC path to that IP  \\xxx.xxx.xxx.xxx\c$ (administrative share)?
0
 

Author Comment

by:Salonge
Comment Utility
Yes, there are dual nic cards on the server, but my laptop only has one.
0
 

Author Comment

by:Salonge
Comment Utility
How would I run a UNC path?
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Are the dual NIC's on server separate or teamed?
Load balanced or redundant on fail? I would have thought that reset when the server was rebuilt.

Is there a secondary DNS that might have the server's IP assigned to another device.
Are you running a Reverse Lookup Zone?
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Click Start then Run for Windows XP or in Win7 just click Start and type:
 \\enter the IP here\c$

c$ is the administrative share. If a window opens then you should be able to determine which computer it is by user profiles, etc. If it's not associated with a computer then you'll see the message network path not found.
0
 

Author Comment

by:Salonge
Comment Utility
When I do the UNC for the IP, I get nothing.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Was really just double checking that yours or any other computer was associated with that IP.

Have you tried pinging the server by name? What IP responds
Have you tried pinging your computer from the server by name or IP?
Did you check to see if you were using a DNS Reverse Lookup Zone.
0
 

Author Comment

by:Salonge
Comment Utility
Have you tried pinging the server by name? What IP responds
Pinged the server by name and the correct new IP responds

Have you tried pinging your computer from the server by name or IP?
Pinged my computer by name and the correct IP responds

Did you check to see if you were using a DNS Reverse Lookup Zone.
I looked in our DNS Reverse Lookup Zone and the bogus IP PTR is my computer.
0
 

Author Comment

by:Salonge
Comment Utility
I deleted the ptr record for the ip address, but I can still ping it.  It just doesn't have anything associated to it.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Narrowing it down... Is your switch/router a Cisco Catalyst?
0
 

Author Comment

by:Salonge
Comment Utility
Yes, it is.  Also I ran a scan on the IP and my computer still showed up.  I clicked on my name and told the system to reboot the computer attached to that IP and my computer rebooted.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
Catalyst 6000/6500 series? Any VLAN's?
0
 

Author Comment

by:Salonge
Comment Utility
No VLANS, 3560s
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
OK, I was reading a fixit how-to for the 6000/6500 series ARP-CAM problems.
I'm about the leave for the day...
I was about to ask if you multiple network configurations on the laptop then it just struck me... do you have wireless turned on your latop as well as the NIC?
0
 
LVL 18

Expert Comment

by:Akinsd
Comment Utility
Reset the NIC on your computer.

Open command prompt as administrator
and issue the following commands
netsh int ip reset
netsh winsock reset
Shutdown your computer and try pinging the ip
Turn your computer on and ping again
0
 

Author Comment

by:Salonge
Comment Utility
Ok I will try that in the morning and let you know
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
If all your switches are managed, then ping the IP address.  Find the MAC and start looking at the mac address table until you find out the "last port" that MAC shows on.

Another possibility, is that somebody has configured a switch to do proxy arp and to respond to pings for that address.
0
 

Author Comment

by:Salonge
Comment Utility
I performed the tasks of resetting my nic card and the phantom IP still thinks it is my IP address.  When I shut my computer down, I can still ping it.  When I ran an IP scan on that IP it says it is my computer with a wierd mac address.
0
 

Author Comment

by:Salonge
Comment Utility
I looked at the ARP table in my router and the IP is showing up there and connected to that wierd mac address.
0
 

Author Comment

by:Salonge
Comment Utility
I checked the Mac address in the mac address table on my main switch and it does not show up anywhere.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Can you post the MAC address?
0
 

Author Comment

by:Salonge
Comment Utility
00:22:64:1A:34:6C
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Do you have any HP equipment in your network?  That is a MAC address assigned to HP.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
00:22:64 is a known HP MAC address
0
 

Author Comment

by:Salonge
Comment Utility
We have several HP's in our environment but none with that address.  My mac table does not find this address.  And since this IP thinks it is my computer, this is not my computer's mac address.
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 

Author Comment

by:Salonge
Comment Utility
how can I clear this ip our of my ARP table on my router to see if it finds it again?
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
What type of router do you have?
0
 

Author Comment

by:Salonge
Comment Utility
Cisco 1900 Series
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
I'm curious when you run ipconfig /all on your laptop... how many different connections do you have? Look for the 00:22:64 MAC address under any of the following:
Ethernet Adapter Bluetooth Network Connection
Wireless LAN Adapter Wireless Network Connection
Ethernet Adapter Local Area Connection
Tunnel Adapter Pseudo-Interface
0
 

Author Comment

by:Salonge
Comment Utility
No, I don't.  I have a HP laptop also.  The interesting concept is that put this IP into my computer to prove to some others that there was not an IP conflict on a major server.  This used to be a server IP address.  Even after we formatted the server, we could ping the IP with -a and the server name would show up.  So I put the IP in my laptop and showed them that my computer did not come up with an IP conflict, nor was it getting kicked off.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
clear arp cache

Should clear all arp entries.  I don't think there is a way to clear an indvidual entry.  They should automatically get cleared out after awhile (I think 4-5 minutes) after not being referenced.

Stupid question.  By chance could there be a device mis-configured so that it thinks that IP address is a IP network broadcast IP address?  Example, if you have your IP subnet as 10.10.10.0/24, the IP address 10.10.10.255 is a broadcast address and if you ping it, devices will respond.  They will NOT report a IP address conflict, because they don't for broadcast addresses.
0
 

Author Comment

by:Salonge
Comment Utility
I don't want to clear all right now midday.  I don't know what affect that will have on anything.  

As far as a mis-configured device, I don't think so.
0
 

Author Comment

by:Salonge
Comment Utility
Will rebooting the router, clear the arp cache?
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Yes, but that will cause more problems than just clearing the arp cache.

-->  And since this IP thinks it is my computer, this is not my computer's mac address.

I must have missed this earlier.  Why do you say that the IP address thinks it belongs to your computer?
0
 

Author Comment

by:Salonge
Comment Utility
It thinks it belongs to my computer because for about 10 minutes, it was the static IP of my computer.  I did this because it was initially a server IP address.  We were having problems with the server and someone said it was an IP conflict.  I said it was not, but something else.  We took the server down, reformatted it, and pinged the IP and it pinged successfully, proving that it was another device somewhere, I still disagreed.  Well, I knew that this was a good IP, not in a DHCP scope and no one else has access to do this.  So to prove that the IP was a good IP, i put it in my computer and my computer worked just fine with no IP conflict.  I am just trying to find out why this IP is out here connected to a phantom Mac address.  I believe it is because of something in my router. I don't want this issue to come and haunt me later.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
It stills sounds like DNS cache (ipconfig /flushdns) or ARP entry in the switch/router.
Is there a secondary DNS server where you could check your Forward and Reverse Lookup Zones like we did yesterday on your primary where you had enties with the IP associated to your computer?
0
 

Author Comment

by:Salonge
Comment Utility
No we only have one DNS server.  I saw the entry in the ARP on the router, but nothing in the mac address table.  It is a mystery.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
To make sure I following this.

The address is actually ping'able when you believe that no host has this IP address assigned to it.  Correct?
0
 

Author Comment

by:Salonge
Comment Utility
yes and I can run a IP scan on that IP and find a Mac address.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
So when you go to Network & Sharing Center - LAN Connection - Properties -  IPv4 Properties - Use the following DNS Server Addresses... there is no entry for Alternate DNS Server?
0
 

Author Comment

by:Salonge
Comment Utility
That is correct.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
If you can find a mac address that address has to exist someplace.

You mentioned you had a Cisco 1900 router.  What type of switches do you use?

Is the "mystery" IP address on the same subnet as your normal IP address?
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
I believe the switch is a Cisco Catalyst 3500 series. I'd read a fixit article yesterday on the Catalyst 6000/6500 series ARP-CAM table issues.
0
 

Author Comment

by:Salonge
Comment Utility
Yes the mystery IP is on the same subnet as all my other IP's.  The mac address is a mac address that has nothing connected to it.  We have Cisco Catalyst Switches 3560.  This is the same Mac address that showed up when it thought it was still connected to the server.
0
 

Author Comment

by:Salonge
Comment Utility
So what is the fix or where can I get my hand on the article?
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
This is specific to the 6000\6500 series but references a number of troubleshooting steps for ARP and CAM:
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/71079-arp-cam-tableissues.html
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
So on the 3650 you do a:

     show arp | i "mystery ip address"

Then you do:

      show mac address table | i "mac associated with mystery ip address"

The MAC does not show?
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
Oh, you may want to do a:

show run | i "mystery ip address"

to make sure somebody did not create a static arp entry for that IP address.
0
 

Author Comment

by:Salonge
Comment Utility
So on the 3650 you do a:

     show arp | i "mystery ip address"

Then you do:

      show mac address table | i "mac associated with mystery ip address"

The MAC does not show?

This is correct.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
giltjr,
I'm I accurate in stating that dynamic ARP entries assigned automatically clear/refresh in up to 20 minute intervals (default to 300 seconds) as long as the IP and MAC  address respond to the ARP sync; and static ARP entries manually added are persistent?
0
 

Author Comment

by:Salonge
Comment Utility
Oh, you may want to do a:

show run | i "mystery ip address"

to make sure somebody did not create a static arp entry for that IP address.

I just did this and no information comes back.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
@Lee Ingalls
Static ARP entries are persistent.  They should show up when you do the "show mac address-table" and they should be shown as static/persistent.

Dynamic arp entries should stay in the table as long as the switch sees traffic FROM it within the timeout.  If, after the timeout period, the switch has not seen traffic from that mac, it will (should) be removed.

@Salonge

Do you only have one switch in your environment?

I would say you may need to open a TAC case with Cisco.  Off hand I can't think of a situation where a router/switch would have a arp entry for an IP address that has a mac address associated with it, but not have a mac address entry in the mac address table.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
One of the issues listed in the Catalyst 6000/6500 series ARP-CAM problems is "two entries show for MAC address in the MAC address table"; depending on which version salonge's 3560's ios is -- may have similar issues.

salonge what version is your switches firmware/ios?
I agree that  Cisco TAC may be the way to go... firmware/ios update may resolve.
0
 

Author Comment

by:Salonge
Comment Utility
I have three managed switches in this building.  I did a ping test and got into my switches to look for the mac address.  The mac address showed up on all three switches on different ports.

Something is not right.
0
 
LVL 57

Accepted Solution

by:
giltjr earned 167 total points
Comment Utility
If it were not for the fact that IP address should not exist at all, showing up on multiple switches on different ports would be normal.

Say you had:

HOST1 <--> SW1 <--> SW2 <--> SW3 <--> HOST2

If you did a ping of host2 from host1, and they were in the same IP subnet.  Then:

On SW1 HOST2's address would show up on the port that connects SW1 and SW2.
On SW2 HOST2's address would show up on the port that connects SW2 and SW3.
On SW2 HOST2's address would show up on the port that HOST2 is on..

Of course in your case the IP address is not supposed to exist.  One of those ports should have a "computer" connected to it, while the other port should be the connection between the switches.

All you need to do now is figure out which port has a computer on it.
0
 
LVL 7

Expert Comment

by:Lee Ingalls
Comment Utility
What IOS releases are your Cisco switches at?
0
 
LVL 18

Expert Comment

by:Akinsd
Comment Utility
IP addresses do get stuck in device cache occasionally and sometimes, that's why I recommended resetting your NIC with the netsh commands.

What's strange here is the IP responds to ping even when the device is shutdown. The closest explanation to this puzzle is a device impersonation either via Man-In-The-Middle scenario or spoofing scenario. A device is responding for your computer somehow.

Wireshark may help reveal flow of packets but narrowing the attack down may be a little challenging. It is easier to set up preventative measures than attempting to narrow it down. at least from my perspective.

http://web.securityinnovation.com/appsec-weekly/blog/bid/63269/How-to-Test-for-Man-in-the-Middle-Vulnerabilities


Have you reloaded (rebooted) your switches yet?

Try the following
- It is not likely that alternative address is configured but it doesn't hurt to check either (on your PC and on the server)

- Perform a tracert from computer and/or traceroute from a switch to that IP

- Check if keep alive is disabled on your switch ports

- Find a good time to reboot all your devices.

- Perform updates where possible
0
 

Author Comment

by:Salonge
Comment Utility
Thanks for everyone's help.  I wish I could award everyone the points because you all were so helpful.
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
You can't award everybody 500 points, but you can split the points between multiple people.

You accept one as a answer, award that a portion of the 500 points, then accept others as assisted and award them portion of the 500 points.
0
 

Author Comment

by:Salonge
Comment Utility
Can I still do that if I have closed the issue?
0
 
LVL 57

Expert Comment

by:giltjr
Comment Utility
You can make a request to have the question re-opened and then do it.

Go back up to the top of the page, and just below the box that has your question there is a yellow triangle "Request Attention", click on that and explain what you want to do.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Let’s list some of the technologies that enable smooth teleworking. 
OfficeMate Freezes on login or does not load after login credentials are input.
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now