ARPing from all over the network

What we have are ARP packets from multiple MAC address ARPing for and addess of 0.11.120.102.   I can see these packets on Many Switches.  It is alwatys an ARP for the address above.  It seems to be comming from everywhere.  I have been unable to locate the cause.
The switches lable this address as a martian destination in the logs.
psfcAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Henk van AchterbergSr. Technical ConsultantCommented:
what kind of device is located at the source mac address? if you trace the source mac you should be able to pinpoint the switchport and identify the device which is asking for this IP.
0
Paul MacDonaldDirector, Information SystemsCommented:
Seems likely you have a machine (maybe more than one, maybe not) that has some malware on it.  Some of the packets you're seeing may be spoofed, so it's hard to be sure.

See what you can glean from your switch logs.  Maybe run Wireshark on your network and see if you can pinpoint a machine (or two) sending out the packets.
0
psfcAuthor Commented:
I see these packets all over the network.  I see them with over 100 different MAC/IP address combinations. I have checked about 15 of these and they seem fine. The MAC associated with the IP addresses are correct in those 15 as well.
The switch logs show only that it sees the packets and they are in the logs because of they Martian Destination in the packets. There are no other entries in the logs across multiple switches.
I though it would be a Malware infection as well. From my capture I have over 100 sending these packets out. pinpointing the offending computer has been very hard.

Since the source appears to be Many, Many MAC addresses, I am stumped.
0
Become a Certified Penetration Testing Engineer

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

Paul MacDonaldDirector, Information SystemsCommented:
It could be something tenacious and well hidden on all 100 computers.  Or it could be something that's spoofing MAC addresses to make eradication difficult.

Do these computers have anything else in common?  A new application?  New updates to something that already existed?
0
Henk van AchterbergSr. Technical ConsultantCommented:
You could set up monitor ports on the uplink of your switches and only capture the rx traffix. This way you can try to locate the switch which is causing this and from the switch go port by port.

if you have cisco switches maybe you could enable port security on your ports and disable when multiple mac addresses are on the port. This way you can also get a clue which device is causing this behaviour.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Craig BeckCommented:
You could have a loop on your network.  Is your STP working correctly?

Also, do you have any routing on your network?  If so, check you've not got any routing loops.
0
Rick_O_ShayCommented:
Can you cut off one switch from the core at a time, maybe outside of production times, to help isolate the source and/or loop?
0
psfcAuthor Commented:
there were a few suggestions.  
Do these computers have anything else in common?  
---- They are all on a flat network
 A new application?
---- None that they have in common
New updates to something that already existed?
----- None
In fact some of the packets that I see are from IP devices like scopes, PDU's, UPS's, I also see it comming from some of the Mac computers.
Is your STP working correctly? as far as I can tell...I only use it on the core switch for the links to the other buildings.
There is no routing on our network.
We use a firewall using Port forward to get traffic onto some of our VLANS.
0
Rick_O_ShayCommented:
If it is a flat network then you will see all ARPs on every port in the network because they are broadcasts and you have no containment in place.
Your devices will all be arping for the router very often. Is it possible the devices reporting the error are interpreting your gateway router's address backwards or something?
I don't think I understand how you have VLANs but no routing?
Does your router/firewall have any kind of proxy ARP enabled?
0
psfcAuthor Commented:
One VLAN is completely isolated, there is no route. The other one we currently use is access with Port forwarding on a small Firewall appliance. We do not use Proxy servers.
0
psfcAuthor Commented:
I udnersatnd that ARPING is going to happen. However I have the specific ARP (above) coming from what looks like many devices.  We do have Broadcast Storm settings turned on to avoid large floods on backbone switches
0
Rick_O_ShayCommented:
If it is "appearing" to come from all of those devices' MACs it still sounds like something is spoofing those in some way. Unless it is happening often enough to manually isolate it by turning stuff off (switch ports, devices, etc.) it will be hard to troubleshoot.
Is there any particular PC or Application that was added at the time this started?

Do your switches allow filtering to drop packets based on destination IP address? I think you can do that on Avaya/Nortel switches and I know you can on Enterasys gear. If so you could maybe roll that policy out to one switch at a time to isolate where this is coming from.

Another thing that might help if this is MAC spoofing is to lock your user ports to only allow the one MAC that belongs there.
0
psfcAuthor Commented:
This started about 18 months ago stopped for about 3 months and now is active again.

I can filter but filtering does not allow the use of Martian Destinations IE 0.11.120.102 as a IP address.  Locking computers to there office port is something I can only do some 1/2 of our systems, so I have not done that.  

I think I have it down to a single building using ingress mirroring to capture.
0
psfcAuthor Commented:
I have found the culprit.  This was a Remote a/c Power Device. we have about 60 of these on our network. It allows the Students and scientist the ability to power cycle there experiments remotely.  

The one seems to have gathered MAC and IP addresses then spoofed ARP's for that weird address 0.11.120.102.  

So it looked like it was coming from all over the place but was just one very dumb IP device.
Thanks All..
B
0
psfcAuthor Commented:
Forgot to say that all this happened without malware.
0
psfcAuthor Commented:
Both of these help me get on the right track.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Networking Protocols

From novice to tech pro — start learning today.