Avatar of HunterIT
HunterIT
Flag for United States of America asked on

ARP Storm Taking Down Default Gateway

We have been experiencing a problem in our local network where the default gateway is being taken down due to what appears to be an ARP storm.

Originally, the default gateway was set to a Cisco 2851 router that routed traffic between several VLANs and had its default route set to our Cisco ASA router. Both the 2951 and the ASA were connected to a Cisco 2560 switch.

When the outage occurred, we lost all routing from the 2851 although we could still access it via Telnet. Clearing the ARP cache would instantly bring all functionality back. We saw a large amount of ARP requests coming in (thousands per minute) and the routing would go back down within about 15 minutes.

To test, we changed the default gateway (set by DHCP) to the ASA router. We experienced the same behaviour of ARP traffic and it would take down the internal interface of the ASA. Clearing ARP instantly brought all functionality back.

We also tried setting up a temporary internet gateway using a Cradlepoint router hooked to a Verizon aircard. It was connected through an intermediant HP switch that was connected to the 2560 switch. After an hour or so, the Cradlepoint was overwhelemed and also went down.

A little more information: We experienced this behaviour two days in a row. Communication inside the same subnets worked fine. Routing would go down around 9:30 AM each day and everything would settle down and become stable around 4:30 pm.

We think the problem is originating from a laptop and only starts happening when the employee arrives to work and then it stops when the employee leaves with their laptop.

Is there any other likely cause to this problem? If it is a laptop, what is the best way to handle this problem? We can wait until it starts happening again on Monday and disconnect switches and ports until we identify the culprit. However, I'd like to prevent any more downtime.

Thanks in advance.
Network SecurityNetwork Management

Avatar of undefined
Last Comment
rauenpc

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
Bradley Fox

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
rauenpc

Dynamic arp inspection combined with dhcp snooping will mitigate this scenario, although it can be a lot of work to implement depending on you environment. Bottom line is that you need to find the source of the arps and remove it. I would guess a faulty nic or more likely a virus.
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes