Have to clear arp cache to get on internet
We have a network of about 50 users and 5 servers. We also have a sonicwall firewall connected to a cisco 1700 which connects to our T1. We have two switches, one HP and one Netgear.
We had a power outage that lasted about 4 hours. We were able to shut everything down gracefully. When everything came back up, we noticed intermittent issues with connectivity. We noticed that one of our terminal servers was not responding to outside connections and our website could not be accessed from the outside. At times when the servers are not responding to the outside, they cannot bring up a web page or even ping our default gateway. I have found that if I clear the arp cache on the box having the issue, everything is fine.
This also happens on some workstations. After a period of inactivity, the workstations will not be able to get to the internet. When this happens, I cannot ping the gateway (sonicwall firewall). If i clear the arp cache, everything works. For now I have created static arp entries in the firewall for the webserver and terminal server and they have been working fine. Obviously I can' do this for all workstations.
I called sonicwall and they were no help.
We had a power outage that lasted about 4 hours. We were able to shut everything down gracefully. When everything came back up, we noticed intermittent issues with connectivity. We noticed that one of our terminal servers was not responding to outside connections and our website could not be accessed from the outside. At times when the servers are not responding to the outside, they cannot bring up a web page or even ping our default gateway. I have found that if I clear the arp cache on the box having the issue, everything is fine.
This also happens on some workstations. After a period of inactivity, the workstations will not be able to get to the internet. When this happens, I cannot ping the gateway (sonicwall firewall). If i clear the arp cache, everything works. For now I have created static arp entries in the firewall for the webserver and terminal server and they have been working fine. Obviously I can' do this for all workstations.
I called sonicwall and they were no help.
ASKER
I thought about that, but something doesn't make sense. When I am on a workstation that cannot get on the internet, it still has the correct arp entry for the gateway. But, it cannot ping it. If I look at the arp table on the firewall at that time, there is no entry for that workstation. If I clear the arp cache on the workstation, it can then connect to the internet. When I go back and look at the arp entry for the gateway, it is the same as before, the correct mac address for the gateway is listed.
It seems like the firewall loses it's arp entry for a workstation, then it has to have a little kick in the but to create a new one and allow connectivity.
Like right now, if I go into the firewall and delete an arp entry for a particular workstation, then that machine will not be able to get to the internet unless I clear the arp cache on that machine. Then, one gets created in the firewall.
It seems like the firewall loses it's arp entry for a workstation, then it has to have a little kick in the but to create a new one and allow connectivity.
Like right now, if I go into the firewall and delete an arp entry for a particular workstation, then that machine will not be able to get to the internet unless I clear the arp cache on that machine. Then, one gets created in the firewall.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Ok, this is weird.
I tried what you said and you can see my ip address sending an echo request to the correct mac address and getting nothing back.
But, the strange thing is, there are a huge amount of packets coming from the sonicwall that say "arp request who is 192.168.1.7 tell 192.168.1.1"
192.168.1.7 is an old server that is no longer on the network. I confirmed that I cannot ping it.
192.168.1.1 is the sonicwall. Why would the firewall be looking for the mac address of a server that is no longer in service?
I tried what you said and you can see my ip address sending an echo request to the correct mac address and getting nothing back.
But, the strange thing is, there are a huge amount of packets coming from the sonicwall that say "arp request who is 192.168.1.7 tell 192.168.1.1"
192.168.1.7 is an old server that is no longer on the network. I confirmed that I cannot ping it.
192.168.1.1 is the sonicwall. Why would the firewall be looking for the mac address of a server that is no longer in service?
can you post the capture file here? WARNING: It may contain information you don't want to show here!!
Regarding your server: Sombody is trying to access the IP address 192.168.1.7 through the firewall, that's the reason why it is trying to get a MAC address. See your logs for connections to that IP address. Anyway, the fact that you firewall does not answer the ping request UNLESS you enter a static ARP entry, shows, that there is a problem with your firewall.
One idea. Do you have NAT configured for the machine you did the test on? Maybe the firewall thinks it has to translate that IP to 192.168.1.7, does that and then tries to get a MAC address. Just an idea....
Regarding your server: Sombody is trying to access the IP address 192.168.1.7 through the firewall, that's the reason why it is trying to get a MAC address. See your logs for connections to that IP address. Anyway, the fact that you firewall does not answer the ping request UNLESS you enter a static ARP entry, shows, that there is a problem with your firewall.
One idea. Do you have NAT configured for the machine you did the test on? Maybe the firewall thinks it has to translate that IP to 192.168.1.7, does that and then tries to get a MAC address. Just an idea....
ASKER
I looked through the connections monitor and could not find any connections to that ip address.
I also waited awhile and captured again. and the result was the same. It is sending gobs of arp broadcasts for a server that is no longer on the network.
I also unplugged the firewall and made sure I could not ping it. Hopefully that confirms nothing else has that ip address.
Attached is the capture file. It is only about 30 seconds worth of capture, but has tons of broadcasts.
I don't think there is anything in there that I don't want public. But if you think there may be, let me know.
Thanks.
capture.txt
I also waited awhile and captured again. and the result was the same. It is sending gobs of arp broadcasts for a server that is no longer on the network.
I also unplugged the firewall and made sure I could not ping it. Hopefully that confirms nothing else has that ip address.
Attached is the capture file. It is only about 30 seconds worth of capture, but has tons of broadcasts.
I don't think there is anything in there that I don't want public. But if you think there may be, let me know.
Thanks.
capture.txt
.cap file would be easier, as I can load that in Wireshark and do some filtering.. Can you post the .cap as well?
I did check the file. The ICMP echo request looks good (IP o.k., MAC o.k.). The ARP "flooding" for 192.168.66.7 is really strange, especially the frequency of the requests. The firewall is really doing something terribly wrong. There is no evidence that there is another device with the same IP on the net. I think its a problem with the SonicWall. Either config problem, or some strange hardware problem.
ASKER
the frequency of the ARP requests for 192.168.66.7 is for sure not good. Regular ARP request would not appear at that rate. Send the capture file to SonicWall support and request an explanation (maybe in text and cap format).
ASKER
Your suggestion to sniff traffic pointed me in the right direction. I finally found that the sonicwall had been configured with .7 as the syslog server. When I took that entry out, everything returned to normal. It was sending out so many arp requests for that decommissioned server that it was messing with everything else. Why it waited until now to freak out is a mystery.
Thanks for the help.
Thanks for the help.
>decommissioned server that it was messing with everything else. Why it waited until >now to freak out is a mystery.
I can explain that :-)
The SonicWall still had a MAC address in the ARP cache for 192.168.66.7 and this entry never expired, as you constantly sent logs to that IP via syslog. syslog uses UDP and does not need any response from the syslog server. When you rebooted the firewall, it lost it's ARP entry and then freaked out while trying to get the MAC address of the syslog server.
Cheers
Kurt
I can explain that :-)
The SonicWall still had a MAC address in the ARP cache for 192.168.66.7 and this entry never expired, as you constantly sent logs to that IP via syslog. syslog uses UDP and does not need any response from the syslog server. When you rebooted the firewall, it lost it's ARP entry and then freaked out while trying to get the MAC address of the syslog server.
Cheers
Kurt
I had the exactly same problem. Called Sonicwall support. Via remote session, the firmware on the Sonicwall was updated. Problem solved. They "knew" the problem when I explained to them.
Scott
Scott
ASKER
In my case the firmware was already the latest version.
Remove the static entires and try to re-create the problem. Then check the arp cache for the IP address of your gateway (arp -a). If you see another MAC address than the one of your gateway, you know what's causing the problem. Then you need to find that device. A first hint will be the frist three bytes of the MAC address, which is a vendor code.
http://www.iana.org/assignments/ethernet-numbers