This new podcast puts you inside the minds of leading white-hat hackers and security researchers. Hosts Marc Laliberte and Corey Nachreiner turn complex security concepts into easily understood and actionable insights on the latest cyber security headlines and trends.
Diagnosing a broadcast storm?
Hello experts,
We are having a strange occurrence on our network that I'm hoping to get some help in diagnosing. We are having random computers "lock up" where we can not even ctr-alt-del, they need to be hard rebooted. All computers have been scanned with Malwarebytes, Vipre, and Combofix with no threat results. We have rewired the network and replaced all switches to ensure no loops. Also, STP is enabled. I have been trying to determine what the source is through packet captures.
When running packet captures there were 4 devices sending out a constant ARP request for the two IPs of 10.0.0.202 and .203. These two IPs do not seem to exist anywhere on the network. They are not in a DHCP scope and return no pings, and I can not find anything with them statically assigned. This network contains one router and multiple switches, all on the same 10.0.0.0/24 subnet, there are no VLANs. Three of the four devices sending out these constant ARP requests were Windows 7 PCs. I have since disconnected them from the network, and the issues seem to have gone away, for now.
However, when running a capture I still see the DNS Server (.100), running 2012 R2 is still sending out ARP requests to the same IPs, constantly. I am not sure what is causing this, any help is greatly appreciated. Thanks in advance.
00:28:13.040905 ARP, Request who-has 10.0.0.203 tell 10.0.0.100, length 46
00:28:13.040951 ARP, Request who-has 10.0.0.202 tell 10.0.0.100, length 46
arp.JPG
We are having a strange occurrence on our network that I'm hoping to get some help in diagnosing. We are having random computers "lock up" where we can not even ctr-alt-del, they need to be hard rebooted. All computers have been scanned with Malwarebytes, Vipre, and Combofix with no threat results. We have rewired the network and replaced all switches to ensure no loops. Also, STP is enabled. I have been trying to determine what the source is through packet captures.
When running packet captures there were 4 devices sending out a constant ARP request for the two IPs of 10.0.0.202 and .203. These two IPs do not seem to exist anywhere on the network. They are not in a DHCP scope and return no pings, and I can not find anything with them statically assigned. This network contains one router and multiple switches, all on the same 10.0.0.0/24 subnet, there are no VLANs. Three of the four devices sending out these constant ARP requests were Windows 7 PCs. I have since disconnected them from the network, and the issues seem to have gone away, for now.
However, when running a capture I still see the DNS Server (.100), running 2012 R2 is still sending out ARP requests to the same IPs, constantly. I am not sure what is causing this, any help is greatly appreciated. Thanks in advance.
00:28:13.040905 ARP, Request who-has 10.0.0.203 tell 10.0.0.100, length 46
00:28:13.040951 ARP, Request who-has 10.0.0.202 tell 10.0.0.100, length 46
arp.JPG
Asked:
Who is Participating?
Experts Exchange Solution brought to you by
Enjoy your complimentary solution view.
Get every solution instantly with premium.
Start your 7-day free trial.
I wear a lot of hats...
"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.
LVL 25
And these 2 IP's also not show up in other traffic except for the ARP requests? Seems as there was something but it's gone now ... Just need to be sure there's really nothing hiding behind these IP-addresses (they could be firewalled or something).
you need to know the hostname of IP 10.0.0.100 by the following command from Windows CMD :
ping -a 10.0.0.100
ping -a 10.0.0.100
Could you post the capture's details of these ARP requests, maybe they shed more light on the issue... Also, maybe the IP-addresses belong to something that was on the network a while ago, could it be network devices? Maybe try to get more info with an ARP viewer.
A ping isn't always reliable to find out if a device is online or not, pings can be dropped by a firewall present on the device. Maybe a port scan?
A ping isn't always reliable to find out if a device is online or not, pings can be dropped by a firewall present on the device. Maybe a port scan?
Here are the details from two packets. Would you be alright with me sending you a PM with full packet capture?
Capture1.JPG
Capture2.JPG
Capture1.JPG
Capture2.JPG
Hi, yeah sure, no problem to send me the packet capture by PM... Might take a little time before I can take a look at it though
Can you check if these ip-addresses are not configured in DNS statically (I know you said you checked, just making sure).
Could you perform a nmap scan of the two ip-addresses to see if they react, it might be it's nothing but better check all the points...
How big is your network? Maybe time to look into broadcast domains/vlans maybe?
Can you check if these ip-addresses are not configured in DNS statically (I know you said you checked, just making sure).
Could you perform a nmap scan of the two ip-addresses to see if they react, it might be it's nothing but better check all the points...
How big is your network? Maybe time to look into broadcast domains/vlans maybe?
I sent you PM with packet captures and NMAP info.
- The network is small, about 15 computers and 5 printers.
- All computers are Windows 7, most are domain joined, some are not on purpose
- Cisco Meraki router
- Cisco Meraki switches, with a couple offices with new gigabit TPlink 5 port switches, with 2 computers connecting to them and possible a printer, cabling back to main Meraki switches
- One on site domain controller, which is a VM running on ESXi 5.5, The VM is server 2012 R2 hosting file server, DNS, and AD.
- This DC communicates across a site to site VPN to another DC, but the issues are only occurring locally, not at the second site.
- Yesterday I had run packet captures and there were 4 computers (one being the server) sending out these arp requests to 10.0.0.202 and .203. I removed the 3 computers from the network, then the only device left sending ARP requests for these 2 IPs was .100, which is the DNS server.
- I then selected a random computer and assigned it statically 10.0.0.202
- Now on new packet captures I no longer see ARP requests for .202, but still .203
- In addition, is looks like now another device is also ARP requesting for .203 as well in addition to the server.
- The network is small, about 15 computers and 5 printers.
- All computers are Windows 7, most are domain joined, some are not on purpose
- Cisco Meraki router
- Cisco Meraki switches, with a couple offices with new gigabit TPlink 5 port switches, with 2 computers connecting to them and possible a printer, cabling back to main Meraki switches
- One on site domain controller, which is a VM running on ESXi 5.5, The VM is server 2012 R2 hosting file server, DNS, and AD.
- This DC communicates across a site to site VPN to another DC, but the issues are only occurring locally, not at the second site.
- Yesterday I had run packet captures and there were 4 computers (one being the server) sending out these arp requests to 10.0.0.202 and .203. I removed the 3 computers from the network, then the only device left sending ARP requests for these 2 IPs was .100, which is the DNS server.
- I then selected a random computer and assigned it statically 10.0.0.202
- Now on new packet captures I no longer see ARP requests for .202, but still .203
- In addition, is looks like now another device is also ARP requesting for .203 as well in addition to the server.
Very curious issue, the arp will continue to ask until it has an answer so it must somehow still receive information about it. I will look at everything you send me first thing tomorrow. Today I got held up in some meetings I'm afraid. Thanks for your patience!
What you seem to be seeing in the scan is a lot of gratuitous arp requests, the curious thing is that it seems the ip-address is not in use (anymore)... I would definitely check out the arp caches on your servers to see if any duplicate mac-addresses are present, just to be on the safe side.
What you also could do is either delete the arp ip-address from the arp cache on all impacted devices or definitely start with that DNS server. You can do this with either "arp -d 10.0.0.203" or the entire cache "netsh interface ip delete arpcache"
See if it stops the annoying arp broadcast ...
Also, try using something like arpwatch to watch arp changes on your network. I'm still wondering where this ip-address came from, how it ended up in the arpcache ...
What you did for the other ip-address so it went away you could try again...
Seems like there was something on your network and it's now gone ... Definitely something to watch for, don't want to sound alarmist, could just be something harmless but better safe than sorry.
What you also could do is either delete the arp ip-address from the arp cache on all impacted devices or definitely start with that DNS server. You can do this with either "arp -d 10.0.0.203" or the entire cache "netsh interface ip delete arpcache"
See if it stops the annoying arp broadcast ...
Also, try using something like arpwatch to watch arp changes on your network. I'm still wondering where this ip-address came from, how it ended up in the arpcache ...
What you did for the other ip-address so it went away you could try again...
Seems like there was something on your network and it's now gone ... Definitely something to watch for, don't want to sound alarmist, could just be something harmless but better safe than sorry.
Experts Exchange Solution brought to you by
Your issues matter to us.
Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.
Start your 7-day free trial
I can't see a broadcast storm causing machines to hang completely. The easy way to test would be to just unplug the patch cable and see if it recovers. If not, I doubt it's a storm causing it.
You could configure broadcast suppression on the switches to see if that stops it though, or just leave one of the machines disconnected for a week or so to see if it suffers the same issue.
You could configure broadcast suppression on the switches to see if that stops it though, or just leave one of the machines disconnected for a week or so to see if it suffers the same issue.
All,
We ended up disconnecting a number of computers that were sending out the broadcasts, they have been disconnected for a week now and the issues have gone away (also I had assigned statically the IPs they were looking for to other devices). We are now connecting one at a time, and waiting a day in between to find the culprit. Very slow and painful task, but it looks like the only way. For now everything is working fine though, thanks for the help.
We ended up disconnecting a number of computers that were sending out the broadcasts, they have been disconnected for a week now and the issues have gone away (also I had assigned statically the IPs they were looking for to other devices). We are now connecting one at a time, and waiting a day in between to find the culprit. Very slow and painful task, but it looks like the only way. For now everything is working fine though, thanks for the help.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis
From novice to tech pro — start learning today.
Question has a verified solution.
Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.
Have a better answer? Share it in a comment.
Experts Exchange Solution brought to you by
Enjoy your complimentary solution view.
Get every solution instantly with premium.
Start your 7-day free trial.