Solved

arp flooding

Posted on 2003-11-07
36
12,065 Views
Last Modified: 2013-11-30
hello,

we are experiencing a problem with our network.

we are being overrun with arp requests. the isp says there is no unusual traffic on the line.
does any one know where to begin looking for this problem?

thanks
0
Comment
Question by:tryonix
  • 11
  • 7
  • 5
  • +6
36 Comments
 
LVL 8

Expert Comment

by:ViRoy
ID: 9704685

first off you will need to identify where they are coming from. sometimes a misconfigured computer can generate these otherwise it might be an attack.
use a packet sniffer and identify the source IP.

if you determine its coming from an internal computer then simply work with that problem pc.
if it is coming from external then take steps to block these such as a firewall or a IDS (intrusion detection system)

and it should just about be that simple.
   1- find whos doing it.
   2- fix it or block it.
0
 
LVL 79

Assisted Solution

by:lrmoore
lrmoore earned 165 total points
ID: 9704774
You are most likely suffering from infected systems on your network.
Infections of Welchia, Nachia, and MSBLAST present these same symptoms:

http://securityresponse.symantec.com/avcenter/venc/data/w32.welchia.worm.removal.tool.html
http://securityresponse.symantec.com/avcenter/venc/data/w32.blaster.worm.removal.tool.html
0
 
LVL 9

Expert Comment

by:TooKoolKris
ID: 9704790
You can shut off ARP replies to unlisted domains/ips on your router.
0
 
LVL 9

Expert Comment

by:TooKoolKris
ID: 9704812
Hey, I just remembered something. Are you patched for the Code Red II worm?
0
 
LVL 9

Expert Comment

by:TooKoolKris
ID: 9704879
Actually I need to clarify my first answer a little. A router performing Directed ARP procedures must filter the propagation of ARP Request packets to constrain the scope of potential "ARP floods" caused by misbehaving routers or hosts, and to terminate potential ARP loops that may occur during periods of routing protocol instability or as a result of inappropriate manual configurations. To control the propagation of an "ARP flood", a router performing Directed ARP procedures could limit the number of identical ARP Requests (i.e., same Source IP address and same Target IP address) that it would forward per small time interval (e.g., no more than one ARP Request per second). Forwarding of ARP Request packets introduces the possibility of ARP loops. The procedures used to control the scope of potential ARP floods may terminate some ARP loops, but additional procedures are needed if the time required to traverse a loop is longer than the timer used to control ARP floods.

The thing is simple ARP floods can be eliminated if routers never forward ARP Requests that were addressed to a link-level broadcast address. If your ISP is saying nothing is on the line my guess is that they mean "their side" of the line. So you may have an internal problem. Check and make sure your routers are acting as they're supposed to and make sure any server running IIS is patched for Code Red.
0
 

Author Comment

by:tryonix
ID: 9705079
i scanned all the computers on the network and none of them are infected.
i think i localized the problem,
i took all the computers off of the network , shut down the server and plugged my laptop
into the dsl router. so the only machine connected to the dsl router is my laptop. i am still getting massive arp requests 2000 - 5000 per minute. when i unplug the dsl line the requests stop. yet the isp still says there is no unusual traffic on the line. any ideas?
0
 

Author Comment

by:tryonix
ID: 9705129
oh and the arp broadcasts coming from the dsl router  are for every ip in our range.

thanks
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 9705311
I'll say it again. That is the exact symptoms of the Welchia and MSBlast infection.
The infected hosts are scanning every IP in the subnet looking for more hosts to infect.
0
 
LVL 7

Expert Comment

by:philby11
ID: 9705464
when you scanned for virus did you disable the system restore on all the PC's that are Win ME & XP?
Also have you tried running Spybot S & D over the network to see what malware is running?
0
 
LVL 1

Expert Comment

by:Scott_V
ID: 9706121
Check your subnet mask on the router...  Make sure its the same as the rest of your subnet.

-Scott
0
 
LVL 18

Assisted Solution

by:chicagoan
chicagoan earned 165 total points
ID: 9707387
Normail  arp requests are broadcasts, they must remain in the collision domain they originated in.
If the MAC address of the arp request corresponds to your DSL device I would check the ethernet setup on it's connection to your switch. A speed / duplex mismatch or poor cabling could cause communications problem. If you have a more sophisticated switch, I would make sure the DSL is plugged into an uplink port or a port which is expecting multiple hosts. If you have any hubs connected to your switch, try unplugging them.
I think it's unlikely you're suffering from a directed arp attack.

If this doesn't help, post your hardware models and configs (edit to remove site specific info) and a sample packet.
0
 

Author Comment

by:tryonix
ID: 9717335
i couldnt find any virus anywhere. i tried everything u guys said. still being flooded.

any more ideas?
0
 
LVL 18

Expert Comment

by:chicagoan
ID: 9717709
You're going to have to get a decent sniffer with reporting capabilities on the collision domain with the router so that you can look at aggregated statistics.

You can use a simple sniffer like ethereal and try to acertain the source IP of the broadcasts, but you'll need something like CA's sniffer pro to paint the picture for you.

Try this if nobody will pop for a decent sniffer:
cheap sniffer:
A CLEAN simple build of 2000 or XP with all windows updates, no office - no nuthin' else.
Ethereal and windcap.

disconect everything from the network
put just your sniffer and the dsl on a HUB and watch the activity.
If you're getting flooded either your ISP is full of cr@p and you're the victim of a denial of service attack or your DSL router is whacked (misconfigures, bad ethernet interface or somthing)
if not - bring you machines up at ten minute intervals and start capturing packets when they've been up for a while
 
 
0
 

Author Comment

by:tryonix
ID: 9771554
i just found out that we share our dsl with 4 other companies. would a virus on one of thier systems cause the arp flooding on ours?

0
 
LVL 79

Expert Comment

by:lrmoore
ID: 9771686
Absolutely...
0
 
LVL 8

Expert Comment

by:ViRoy
ID: 9772182
actually it would depend on how they "Share"
is there 1 DSL modem for all 4 companies?

i used to work for a DSL provider and am pretty sure you are not allowed to share a single business connection between 4 business, that is illegal. maybe all 4 business have DSL and share from the same physical trunk, but not the exact same connection.

either way your not gonna be able to tell by looking at it from the "outside"
you WILL need to capture a few packets and that will INSTANTLY tell you where the arp requests are origionating from. and then you can tracert the origionating host to further track it down.



0
 

Author Comment

by:tryonix
ID: 9773928
each company has its own dsl modem , but they are on the same trunk.

i didnt do a traceroute yet but ethereal lists the source ip as our router.

so if one of them has a virus ,  it would cause the arp flooding on ours?


thanks
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 9773973
If they have their own modems, they should not be affecting yours.
Even though the source packets look like they are coming from your router, the culprit is most likely still inside your network.
You need a sniffer...
Try a demo copy of Lanhound, http://www.extralan.co.uk/Downloads/Lanhounddownload.htm or use ethereal http://www.ethereal.com
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 8

Expert Comment

by:ViRoy
ID: 9774682

thats odd it looks as origionating from the router, routers will not forward ARP's so it shouldnt be coming from an external source.

first lets try to determine the router as a problem or not.
im wondering what the "discovery method" is for the router... try logging into the interface and search for anything relating to ARP. also look for anything about "IP Forwarding" and make sure it is NOT enabled.

0
 

Author Comment

by:tryonix
ID: 9784745
i have narrowed it down to either a bad router at the isp. or another company on the same trunk that has a virus on there systems.

it is definatly not a problem with our network. i unplugged everything from the dsl modem/router except for the dsl connection and a laptop with a fresh install of xp and etherreal. and we are still getting thousands of arps per second. when i disconnect the dsl the requests stop so its not our router that is doing it. it is definatly external. the requests are for every ip except the one the laptop is using. i can't see where the request are coming from all ethereal tells me is that they are arp requests and they are in the form -- who has xxx.xxx.xxx.xxx -- tell router's ip. and the source ip is the router. ip forwarding is not enabled, the router dosn't let me block arps.


we need more bandwith any way so we are going with a T1, we will be getting a new router and a new range of ip's and we will no longer be sharing a trunk with anybody else. This should solve the problem. i hope



0
 
LVL 8

Expert Comment

by:ViRoy
ID: 9788687

wow thats pretty wild, sounds like their equipment is not setup properly. i believe if you call your isp they will take care of you, business connections usually get a much higher priority than residential. also, if you do call them to support this, be a lil picky and upset about the performance you have recieved and you should be offered some kind of compensation.
0
 

Author Comment

by:tryonix
ID: 9791494
yeah,  the weirdest thing is the arp requests are only for our unused ips.

i called them up. all i got from them is - we can ping the router so everything is fine.

that is as far as they would go.

does anyone have any other ideas?

0
 
LVL 8

Accepted Solution

by:
ViRoy earned 170 total points
ID: 9798859

well.... we will have to prove that they need to do more.

first off, what exact bandwidth are you paying for? 768k?
next go to http://www.dslreports.com/stest and see what your really getting.
IF you get results at half or less of what your paying for then i would call them back and tell them you "have been able to access the internet all along, but performance is unacceptable as its only half of what im paying for. i think i might have discovered the problem but am not sure and was wondering if you could get this taken care of."

then hopefully he'll ask you what you discovered and open a trouble ticket for you.
then checkup every 24 hours.
(IMPORTANT when explaining, be general about saying what you found, let them find the fine details cause thats their job... that and i hate it when customers call and try to tell me they know whats goin on when they dont know for sure) i would say something along the line of " i ran a network sniffer and noticed an enormous amount of ARP packets." and leave it at that.

0
 

Author Comment

by:tryonix
ID: 9814304
thanks for the help guys
0
 
LVL 8

Expert Comment

by:ViRoy
ID: 9818562

so did it ever get resolved?
0
 

Author Comment

by:tryonix
ID: 9819369
no we are gonna go with a t1
0
 

Author Comment

by:tryonix
ID: 9819411
has not been resolved still getting flooded. with ethereal i am seeing almost 90 percent of my traffic is arp traffic. at least 1000 arps a second. i give up. going with a t1 and new ips. should resolve problem (i hope)
0
 
LVL 8

Expert Comment

by:ViRoy
ID: 9821574
sorry to hear that
T1's are pretty expensive but if you can utilize the bandwidth your likely to increase revenue

0
 

Expert Comment

by:ESQuicksall
ID: 10311713
Hi,

Did the T1 resolve this?

We are also getting ARP flooding but it is in the lab where we don't have an outside connection. Ours is comming from a W2K system.
0
 

Author Comment

by:tryonix
ID: 10333607
yes the T1 resolved it.

we are no longer getting flooded.
0
 

Expert Comment

by:ESQuicksall
ID: 10333703
We are getting ARP flooding too but it is on a direct connect on a 1Gig connection from a Windows 2000 computer. We only get it when running at  high speed and with the Microsoft iSCSI initiator. Has anyone ever heard of such a thing?
0
 
LVL 18

Expert Comment

by:chicagoan
ID: 10335922
sounds like a different question  ESQuicksall...
0
 

Expert Comment

by:ESQuicksall
ID: 10335951
Yea, I re-submitted it under "ARP flooding with Microsoft iSCSI initiator".
0
 

Expert Comment

by:barnster123
ID: 21018635
I get tons of these in a tcp dump, These are stright offf the broadband network and nothing to do with my I.P or Gateway I.p so why the hell do I get them.

23:26:59.078518 arp who-has 82-40-178-39.cable.ubr10.nmal.blueyonder.co.uk tell
82-40-178-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.099728 arp who-has 80-195-182-175.cable.ubr10.nmal.blueyonder.co.uk tel
l 80-195-182-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.158508 arp who-has 77-102-232-205.cable.ubr10.nmal.blueyonder.co.uk tel
l 77-102-232-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.285153 arp who-has 80-195-182-106.cable.ubr10.nmal.blueyonder.co.uk tel
l 80-195-182-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.379440 arp who-has 77-102-232-207.cable.ubr10.nmal.blueyonder.co.uk tel
l 77-102-232-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.407071 arp who-has 82-40-178-168.cable.ubr10.nmal.blueyonder.co.uk tell
 82-40-178-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.468876 arp who-has 77-102-232-208.cable.ubr10.nmal.blueyonder.co.uk tel
l 77-102-232-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.478590 arp who-has 82-43-213-236.cable.ubr10.nmal.blueyonder.co.uk tell
 82-43-208-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.494445 arp who-has 82-43-208-111.cable.ubr10.newm.blueyonder.co.uk tell
 82-43-208-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.506298 arp who-has 82-40-178-71.cable.ubr10.nmal.blueyonder.co.uk tell
82-40-178-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.630377 arp who-has 77-101-49-233.cable.ubr10.nmal.blueyonder.co.uk tell
 77-101-49-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.653808 arp who-has 82-44-85-165.cable.ubr10.nmal.blueyonder.co.uk tell
82-44-84-1.cable.ubr10.nmal.blueyonder.co.uk
23:26:59.668562 arp who-has 82-44-84-177.cable.ubr10.nmal.blueyonder.co.uk tell
82-44-84-1.cable.ubr10.nmal.blueyonder.co.uk
0
 

Expert Comment

by:ESQuicksall
ID: 21040627
My problem was due to too much going on in the computer. The IP stack was not able to respond to the ARP fast enough and hence another ARP was sent very quickly. We simply fixed the bug in our IP stack.
0
 

Expert Comment

by:ESQuicksall
ID: 21040671
The computer at the "tell" address is the one sending the ARP. It is strange that so many different systems are all sending their ARP in the same second.
0

Featured Post

Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

Join & Write a Comment

Article by: IanTh
Hi Guys After a whole weekend getting wake on lan over the internet working, I thought I would share the experience. Your firewall has to have a port forward for port 9 udp to your local broadcast x.x.x.255 but if that doesnt work, do it to a …
Short answer to this question: there is no effective WiFi manager in iOS devices as seen in Windows WiFi or Macbook OSx WiFi management, but this article will try and provide some amicable solutions to better suite your needs.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now