We help IT Professionals succeed at work.

Diagnosing a broadcast storm?

CCtech
CCtech asked
on
Hello experts,

   We are having a strange occurrence on our network that I'm hoping to get some help in diagnosing. We are having random computers "lock up" where we can not even ctr-alt-del, they need to be hard rebooted. All computers have been scanned with Malwarebytes, Vipre, and Combofix with no threat results. We have rewired the network and replaced all switches to ensure no loops. Also, STP is enabled. I have been trying to determine what the source is through packet captures.

   When running packet captures there were 4 devices sending out a constant ARP request for the two IPs of 10.0.0.202 and .203. These two IPs do not seem to exist anywhere on the network. They are not in a DHCP scope and return no pings, and I can not find anything with them statically assigned. This network contains one router and multiple switches, all on the same 10.0.0.0/24 subnet, there are no VLANs. Three of the four devices sending out these constant ARP requests were Windows 7 PCs. I have since disconnected them from the network, and the issues seem to have gone away, for now.

  However, when running a capture I still see the DNS Server (.100), running 2012 R2 is still sending out ARP requests to the same IPs, constantly. I am not sure what is causing this, any help is greatly appreciated. Thanks in advance.

00:28:13.040905 ARP, Request who-has 10.0.0.203 tell 10.0.0.100, length 46
00:28:13.040951 ARP, Request who-has 10.0.0.202 tell 10.0.0.100, length 46
arp.JPG
Comment
Watch Question

View Solution Only

Zephyr ICTCloud Architect

Commented:
And these 2 IP's also not show up in other traffic except for the ARP requests? Seems as there was something but it's gone now ... Just need to be sure there's really nothing hiding behind these IP-addresses (they could be firewalled or something).
nader alkahtaniInformation security consultant

Commented:
you need to know the hostname of IP 10.0.0.100 by the following command from Windows CMD :
ping -a 10.0.0.100
Zephyr ICTCloud Architect

Commented:
Could you post the capture's details of these ARP requests, maybe they shed more light on the issue... Also, maybe the IP-addresses belong to something that was on the network a while ago, could it be network devices? Maybe try to get more info with an ARP viewer.

A ping isn't always reliable to find out if a device is online or not, pings can be dropped by a firewall present on the device. Maybe a port scan?
CCtech

Author

Commented:
Here are the details from two packets. Would you be alright with me sending you a PM with full packet capture?
Capture1.JPG
Capture2.JPG
Zephyr ICTCloud Architect

Commented:
Hi, yeah sure, no problem to send me the packet capture by PM... Might take a little time before I can take a look at it though

Can you check if these ip-addresses are not configured in DNS statically (I know you said you checked, just making sure).

Could you perform a nmap scan of the two ip-addresses to see if they react, it might be it's nothing but better check all the points...

How big is your network? Maybe time to look into broadcast domains/vlans maybe?
CCtech

Author

Commented:
I sent you PM with packet captures and NMAP info.

- The network is small, about 15 computers and 5 printers.
- All computers are Windows 7, most are domain joined, some are not on purpose
- Cisco Meraki router
- Cisco Meraki switches, with a couple offices with new gigabit TPlink 5 port switches, with 2 computers connecting to them and possible a printer, cabling back to main Meraki switches
- One on site domain controller, which is a VM running on ESXi 5.5, The VM is server 2012 R2 hosting file server, DNS, and AD.
- This DC communicates across a site to site VPN to another DC, but the issues are only occurring locally, not at the second site.

- Yesterday I had run packet captures and there were 4 computers (one being the server) sending out these arp requests to 10.0.0.202 and .203. I removed the 3 computers from the network, then the only device left sending ARP requests for these 2 IPs was .100, which is the DNS server.
-  I then selected a random computer and assigned it statically 10.0.0.202
- Now on new packet captures I no longer see ARP requests for .202, but still .203
- In addition, is looks like now another device is also ARP requesting for .203 as well in addition to the server.
Zephyr ICTCloud Architect

Commented:
Very curious issue, the arp will continue to ask until it has an answer so it must somehow still receive information about it. I will look at everything you send me first thing tomorrow. Today I got held up in some meetings I'm afraid. Thanks for your patience!
Cloud Architect
Commented:
What you seem to be seeing in the scan is a lot of gratuitous arp requests, the curious thing is that it seems the ip-address is not in use (anymore)... I would definitely check out the arp caches on your servers to see if any duplicate mac-addresses are present, just to be on the safe side.

What you also could do is either delete the arp ip-address from the arp cache on all impacted devices or definitely start with that DNS server. You can do this with either "arp -d 10.0.0.203" or the entire cache "netsh interface ip delete arpcache"

See if it stops the annoying arp broadcast ...

Also, try using something like arpwatch to watch arp changes on your network. I'm still wondering where this ip-address came from, how it ended up in the arpcache ...

What you did for the other ip-address so it went away you could try again...

Seems like there was something on your network and it's now gone ... Definitely something to watch for, don't want to sound alarmist, could just be something harmless but better safe than sorry.
some one
Top Expert 2014

Commented:
I can't see a broadcast storm causing machines to hang completely.  The easy way to test would be to just unplug the patch cable and see if it recovers.  If not, I doubt it's a storm causing it.

You could configure broadcast suppression on the switches to see if that stops it though, or just leave one of the machines disconnected for a week or so to see if it suffers the same issue.
CCtech

Author

Commented:
All,

    We ended up disconnecting a number of computers that were sending out the broadcasts, they have been disconnected for a week now and the issues have gone away (also I had assigned statically the IPs they were looking for to other devices). We are now connecting one at a time, and waiting a day in between to find the culprit. Very slow and painful task, but it looks like the only way. For now everything is working fine though, thanks for the help.