Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!
Cisco ASA5505 keeps disconnecting
Hey guys,
I have a Cisco ASA 5505 here connected to a 5510 VPN. All has been running well for quite some time (2yrs) no issues. recently, the last 3 days, I've been getting disconnects. the local Lan is fine, all internal traffic looks good, the ISP modem is good since I have a seperate device plugged into it and it never drops.
- For the past few days after doing a couple hard boots it will work for a bit (time is inconsistant, sometimes 15 mins, sometimes an hr).
- I can clear conn, clear xlate and clear arp to make sure nothing binded there. sometimes it helps after I clear then reload, sometimes it doesn't help at all.
- I've changed External IP, and it seems to help for a little bit. then starts the same thing again.
one thing I am noticing when clearing the connections, when the LAN cannot connect to the internet, it seems there are abnormal amount of outside DNS requests going to the internal IPs, many times each PC have Multiple simultaneous requests. like 10 - 15 per pc so my "sh conn" jump from 15 to easily over 100. I only have 15 devices at the site. maybe 20 with phones connecting to wifi.
I'm not sure if this is a DNS DOS attack from someone inside or outside?
I have tried changing the DNS servers and get the same result. I'm at the point now where I have all devices turned off or disconnected from the network and can't seem to get the ASA internet back up this time at all.
Any all help would be appreciated.
I have a Cisco ASA 5505 here connected to a 5510 VPN. All has been running well for quite some time (2yrs) no issues. recently, the last 3 days, I've been getting disconnects. the local Lan is fine, all internal traffic looks good, the ISP modem is good since I have a seperate device plugged into it and it never drops.
- For the past few days after doing a couple hard boots it will work for a bit (time is inconsistant, sometimes 15 mins, sometimes an hr).
- I can clear conn, clear xlate and clear arp to make sure nothing binded there. sometimes it helps after I clear then reload, sometimes it doesn't help at all.
- I've changed External IP, and it seems to help for a little bit. then starts the same thing again.
one thing I am noticing when clearing the connections, when the LAN cannot connect to the internet, it seems there are abnormal amount of outside DNS requests going to the internal IPs, many times each PC have Multiple simultaneous requests. like 10 - 15 per pc so my "sh conn" jump from 15 to easily over 100. I only have 15 devices at the site. maybe 20 with phones connecting to wifi.
I'm not sure if this is a DNS DOS attack from someone inside or outside?
I have tried changing the DNS servers and get the same result. I'm at the point now where I have all devices turned off or disconnected from the network and can't seem to get the ASA internet back up this time at all.
Any all help would be appreciated.
Do more with
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Does the interface bounce between up/down status?
It could be excessive traffic coming in from outside, but I have seen where a failing network interface would cause this.
If you have a spare interface, I would first try swapping them out. This would at least rule out the possibility.
Also, have you tried changing the network cable? You wouldn't think a cable would go bad, but I have seen on a number of occasions where we had troubleshot a connectivity issue only to find out the the problem was the network cable.
It could be excessive traffic coming in from outside, but I have seen where a failing network interface would cause this.
If you have a spare interface, I would first try swapping them out. This would at least rule out the possibility.
Also, have you tried changing the network cable? You wouldn't think a cable would go bad, but I have seen on a number of occasions where we had troubleshot a connectivity issue only to find out the the problem was the network cable.
heat seems fine, its sitting on top of a cabinet and it's not overly warm in the room.
interfaces show all up.
I have another ASA coming in to replace as I was thinking the outside interface is failing too. but still trying to figure it out until then.
I swapped out cables the other day. it was stable for a bit then started again.
interfaces show all up.
I have another ASA coming in to replace as I was thinking the outside interface is failing too. but still trying to figure it out until then.
I swapped out cables the other day. it was stable for a bit then started again.
Have you looked at how many packets are coming in/out compared to packets going out/in?
Sometimes if there is an extremely large discrepancy between the two you might have something spamming the interface. I have also seen where another computer on the network had a failing nic that was spamming the interface causing similar issues.
Sometimes if there is an extremely large discrepancy between the two you might have something spamming the interface. I have also seen where another computer on the network had a failing nic that was spamming the interface causing similar issues.
Interface Vlan2 "outside", is up, line protocol is up
Hardware is EtherSVI, BW 100 Mbps, DLY 100 usec
Traffic Statistics for "outside":
110203 packets input, 5253188 bytes
4699 packets output, 937682 bytes
1686 packets dropped
1 minute input rate 16 pkts/sec, 757 bytes/sec
1 minute output rate 0 pkts/sec, 73 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 15 pkts/sec, 743 bytes/sec
5 minute output rate 0 pkts/sec, 72 bytes/sec
5 minute drop rate, 0 pkts/sec
Incomming seem quite high compared to outgoing. kinda reflects how I see so many inbound UDP connections coming from ISP DNS address to the local machines. I've since changed the DNS to Googles DNS and my local and get the same result.
Hardware is EtherSVI, BW 100 Mbps, DLY 100 usec
Traffic Statistics for "outside":
110203 packets input, 5253188 bytes
4699 packets output, 937682 bytes
1686 packets dropped
1 minute input rate 16 pkts/sec, 757 bytes/sec
1 minute output rate 0 pkts/sec, 73 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 15 pkts/sec, 743 bytes/sec
5 minute output rate 0 pkts/sec, 72 bytes/sec
5 minute drop rate, 0 pkts/sec
Incomming seem quite high compared to outgoing. kinda reflects how I see so many inbound UDP connections coming from ISP DNS address to the local machines. I've since changed the DNS to Googles DNS and my local and get the same result.
If you are really curious, you could setup a packet sniffer between you internet connection and router to see what the incoming traffic is? you could use something like Wireshark to do the capturing.
Or, you might be able to contact your ISP and see if they have a website you can see that shows the traffic?
Or, you might be able to contact your ISP and see if they have a website you can see that shows the traffic?
yeah I suppose, but If I changed external IPs, how long would it take for them to get on it again? a day or so?
Changing external IP and updating DNS entries can take up to 3 days (standard ISP answer). Normally, it is around 24 hours or sooner. Depends how often DNS providers transfers DNS zone records.
However, you can put a computer inline as a passthrough or use a hub/switch and not have to change the external IP address.
Hub, gives you one broadcast domain so all traffic is seen by all other ports.
A switch would have to have spanned ports configured to see the other traffic not destined for the computer port.
Hub, gives you one broadcast domain so all traffic is seen by all other ports.
A switch would have to have spanned ports configured to see the other traffic not destined for the computer port.
here is an example of what I'm seeing when I try a sh conn. keep in mind this is the google DNS and some internal IPs. I had switched the DNS from my ISP DNS to google for an external DNS flushed the DNS on the local PCs and still had the same issue, so I'm not 100% sure where it's coming from. most cases we have hundreds of these connections showing, and whey I don't see DNS entries like this, the connection is up. so it's got to be related.
11 in use, 100 most used
UDP outside 8.8.4.4:53 inside 192.168.22.51:54329, idle 0:01:44, bytes 156, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50601, idle 0:01:44, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:52147, idle 0:01:45, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:65169, idle 0:01:52, bytes 208, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:54335, idle 0:01:56, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:62276, idle 0:01:57, bytes 164, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:53163, idle 0:01:57, bytes 168, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50954, idle 0:01:59, bytes 168, flags -
Tks for your help guys. I'm going to throw a hub in the mix or configure port mirroring on the ASA to mirror the outside interface and see what I come up with. i'll post up my results.
Tks again for the info, keep the recommendations coming!
11 in use, 100 most used
UDP outside 8.8.4.4:53 inside 192.168.22.51:54329, idle 0:01:44, bytes 156, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50601, idle 0:01:44, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:52147, idle 0:01:45, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:65169, idle 0:01:52, bytes 208, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:54335, idle 0:01:56, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:62276, idle 0:01:57, bytes 164, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:53163, idle 0:01:57, bytes 168, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50954, idle 0:01:59, bytes 168, flags -
Tks for your help guys. I'm going to throw a hub in the mix or configure port mirroring on the ASA to mirror the outside interface and see what I come up with. i'll post up my results.
Tks again for the info, keep the recommendations coming!
Sorry about the delays guys. I've since travelled to the site, (I've been troubleshooting this remotely as the site is 3hrs away and I have 5 other sites I look after), installed a new ASA 5505 box and configured. everything was working till this morning. Connection was solid yesterday till sometime around 3AM when the network started to replicate the same issue again.
I do have my sniffer machine connected with 2 nics so I can see whats going on inside the ASA as well as outside traffic to the ASA.
From what I can see, inside the network, the VPN tunnel seems to be flapping now. I can see it's up, but it's not receiving traffic sporatically. Tx seems to be normal, but after a little bit, the VPN tunnel closes, then reopens again and on goes the cycle.
I'm not seeing anything too odd in wireshark, however my assumption is the same, the connection seems to drop when I start to see alot of UDP connections from outside DNS address to the inside PC addresses as I posted an example above.
I'm going to continue watching traffic to see if I can find any other trends, but as it stands, the site is online using a SOHO router to the ISP.
I do have my sniffer machine connected with 2 nics so I can see whats going on inside the ASA as well as outside traffic to the ASA.
From what I can see, inside the network, the VPN tunnel seems to be flapping now. I can see it's up, but it's not receiving traffic sporatically. Tx seems to be normal, but after a little bit, the VPN tunnel closes, then reopens again and on goes the cycle.
I'm not seeing anything too odd in wireshark, however my assumption is the same, the connection seems to drop when I start to see alot of UDP connections from outside DNS address to the inside PC addresses as I posted an example above.
I'm going to continue watching traffic to see if I can find any other trends, but as it stands, the site is online using a SOHO router to the ISP.
Premium Content
You need an Expert Office subscription to comment.Start Free Trial