Managing application connectivity and security policies can be achieved more effectively when following a framework that automates repeatable processes and ensures that the right activities are performed in the right order.
Cisco ASA5505 keeps disconnecting
Hey guys,
I have a Cisco ASA 5505 here connected to a 5510 VPN. All has been running well for quite some time (2yrs) no issues. recently, the last 3 days, I've been getting disconnects. the local Lan is fine, all internal traffic looks good, the ISP modem is good since I have a seperate device plugged into it and it never drops.
- For the past few days after doing a couple hard boots it will work for a bit (time is inconsistant, sometimes 15 mins, sometimes an hr).
- I can clear conn, clear xlate and clear arp to make sure nothing binded there. sometimes it helps after I clear then reload, sometimes it doesn't help at all.
- I've changed External IP, and it seems to help for a little bit. then starts the same thing again.
one thing I am noticing when clearing the connections, when the LAN cannot connect to the internet, it seems there are abnormal amount of outside DNS requests going to the internal IPs, many times each PC have Multiple simultaneous requests. like 10 - 15 per pc so my "sh conn" jump from 15 to easily over 100. I only have 15 devices at the site. maybe 20 with phones connecting to wifi.
I'm not sure if this is a DNS DOS attack from someone inside or outside?
I have tried changing the DNS servers and get the same result. I'm at the point now where I have all devices turned off or disconnected from the network and can't seem to get the ASA internet back up this time at all.
Any all help would be appreciated.
I have a Cisco ASA 5505 here connected to a 5510 VPN. All has been running well for quite some time (2yrs) no issues. recently, the last 3 days, I've been getting disconnects. the local Lan is fine, all internal traffic looks good, the ISP modem is good since I have a seperate device plugged into it and it never drops.
- For the past few days after doing a couple hard boots it will work for a bit (time is inconsistant, sometimes 15 mins, sometimes an hr).
- I can clear conn, clear xlate and clear arp to make sure nothing binded there. sometimes it helps after I clear then reload, sometimes it doesn't help at all.
- I've changed External IP, and it seems to help for a little bit. then starts the same thing again.
one thing I am noticing when clearing the connections, when the LAN cannot connect to the internet, it seems there are abnormal amount of outside DNS requests going to the internal IPs, many times each PC have Multiple simultaneous requests. like 10 - 15 per pc so my "sh conn" jump from 15 to easily over 100. I only have 15 devices at the site. maybe 20 with phones connecting to wifi.
I'm not sure if this is a DNS DOS attack from someone inside or outside?
I have tried changing the DNS servers and get the same result. I'm at the point now where I have all devices turned off or disconnected from the network and can't seem to get the ASA internet back up this time at all.
Any all help would be appreciated.
Who is Participating?
Experts Exchange Solution brought to you by
Enjoy your complimentary solution view.
Get every solution instantly with Premium.
Start your 7-day free trial.
I wear a lot of hats...
"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.
Does the interface bounce between up/down status?
It could be excessive traffic coming in from outside, but I have seen where a failing network interface would cause this.
If you have a spare interface, I would first try swapping them out. This would at least rule out the possibility.
Also, have you tried changing the network cable? You wouldn't think a cable would go bad, but I have seen on a number of occasions where we had troubleshot a connectivity issue only to find out the the problem was the network cable.
It could be excessive traffic coming in from outside, but I have seen where a failing network interface would cause this.
If you have a spare interface, I would first try swapping them out. This would at least rule out the possibility.
Also, have you tried changing the network cable? You wouldn't think a cable would go bad, but I have seen on a number of occasions where we had troubleshot a connectivity issue only to find out the the problem was the network cable.
heat seems fine, its sitting on top of a cabinet and it's not overly warm in the room.
interfaces show all up.
I have another ASA coming in to replace as I was thinking the outside interface is failing too. but still trying to figure it out until then.
I swapped out cables the other day. it was stable for a bit then started again.
interfaces show all up.
I have another ASA coming in to replace as I was thinking the outside interface is failing too. but still trying to figure it out until then.
I swapped out cables the other day. it was stable for a bit then started again.
Have you looked at how many packets are coming in/out compared to packets going out/in?
Sometimes if there is an extremely large discrepancy between the two you might have something spamming the interface. I have also seen where another computer on the network had a failing nic that was spamming the interface causing similar issues.
Sometimes if there is an extremely large discrepancy between the two you might have something spamming the interface. I have also seen where another computer on the network had a failing nic that was spamming the interface causing similar issues.
Interface Vlan2 "outside", is up, line protocol is up
Hardware is EtherSVI, BW 100 Mbps, DLY 100 usec
Traffic Statistics for "outside":
110203 packets input, 5253188 bytes
4699 packets output, 937682 bytes
1686 packets dropped
1 minute input rate 16 pkts/sec, 757 bytes/sec
1 minute output rate 0 pkts/sec, 73 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 15 pkts/sec, 743 bytes/sec
5 minute output rate 0 pkts/sec, 72 bytes/sec
5 minute drop rate, 0 pkts/sec
Incomming seem quite high compared to outgoing. kinda reflects how I see so many inbound UDP connections coming from ISP DNS address to the local machines. I've since changed the DNS to Googles DNS and my local and get the same result.
Hardware is EtherSVI, BW 100 Mbps, DLY 100 usec
Traffic Statistics for "outside":
110203 packets input, 5253188 bytes
4699 packets output, 937682 bytes
1686 packets dropped
1 minute input rate 16 pkts/sec, 757 bytes/sec
1 minute output rate 0 pkts/sec, 73 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 15 pkts/sec, 743 bytes/sec
5 minute output rate 0 pkts/sec, 72 bytes/sec
5 minute drop rate, 0 pkts/sec
Incomming seem quite high compared to outgoing. kinda reflects how I see so many inbound UDP connections coming from ISP DNS address to the local machines. I've since changed the DNS to Googles DNS and my local and get the same result.
If you are really curious, you could setup a packet sniffer between you internet connection and router to see what the incoming traffic is? you could use something like Wireshark to do the capturing.
Or, you might be able to contact your ISP and see if they have a website you can see that shows the traffic?
Or, you might be able to contact your ISP and see if they have a website you can see that shows the traffic?
yeah I suppose, but If I changed external IPs, how long would it take for them to get on it again? a day or so?
Changing external IP and updating DNS entries can take up to 3 days (standard ISP answer). Normally, it is around 24 hours or sooner. Depends how often DNS providers transfers DNS zone records.
However, you can put a computer inline as a passthrough or use a hub/switch and not have to change the external IP address.
Hub, gives you one broadcast domain so all traffic is seen by all other ports.
A switch would have to have spanned ports configured to see the other traffic not destined for the computer port.
Hub, gives you one broadcast domain so all traffic is seen by all other ports.
A switch would have to have spanned ports configured to see the other traffic not destined for the computer port.
here is an example of what I'm seeing when I try a sh conn. keep in mind this is the google DNS and some internal IPs. I had switched the DNS from my ISP DNS to google for an external DNS flushed the DNS on the local PCs and still had the same issue, so I'm not 100% sure where it's coming from. most cases we have hundreds of these connections showing, and whey I don't see DNS entries like this, the connection is up. so it's got to be related.
11 in use, 100 most used
UDP outside 8.8.4.4:53 inside 192.168.22.51:54329, idle 0:01:44, bytes 156, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50601, idle 0:01:44, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:52147, idle 0:01:45, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:65169, idle 0:01:52, bytes 208, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:54335, idle 0:01:56, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:62276, idle 0:01:57, bytes 164, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:53163, idle 0:01:57, bytes 168, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50954, idle 0:01:59, bytes 168, flags -
Tks for your help guys. I'm going to throw a hub in the mix or configure port mirroring on the ASA to mirror the outside interface and see what I come up with. i'll post up my results.
Tks again for the info, keep the recommendations coming!
11 in use, 100 most used
UDP outside 8.8.4.4:53 inside 192.168.22.51:54329, idle 0:01:44, bytes 156, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50601, idle 0:01:44, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:52147, idle 0:01:45, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:65169, idle 0:01:52, bytes 208, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:54335, idle 0:01:56, bytes 196, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:62276, idle 0:01:57, bytes 164, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:53163, idle 0:01:57, bytes 168, flags -
UDP outside 8.8.4.4:53 inside 192.168.22.51:50954, idle 0:01:59, bytes 168, flags -
Tks for your help guys. I'm going to throw a hub in the mix or configure port mirroring on the ASA to mirror the outside interface and see what I come up with. i'll post up my results.
Tks again for the info, keep the recommendations coming!
Sorry about the delays guys. I've since travelled to the site, (I've been troubleshooting this remotely as the site is 3hrs away and I have 5 other sites I look after), installed a new ASA 5505 box and configured. everything was working till this morning. Connection was solid yesterday till sometime around 3AM when the network started to replicate the same issue again.
I do have my sniffer machine connected with 2 nics so I can see whats going on inside the ASA as well as outside traffic to the ASA.
From what I can see, inside the network, the VPN tunnel seems to be flapping now. I can see it's up, but it's not receiving traffic sporatically. Tx seems to be normal, but after a little bit, the VPN tunnel closes, then reopens again and on goes the cycle.
I'm not seeing anything too odd in wireshark, however my assumption is the same, the connection seems to drop when I start to see alot of UDP connections from outside DNS address to the inside PC addresses as I posted an example above.
I'm going to continue watching traffic to see if I can find any other trends, but as it stands, the site is online using a SOHO router to the ISP.
I do have my sniffer machine connected with 2 nics so I can see whats going on inside the ASA as well as outside traffic to the ASA.
From what I can see, inside the network, the VPN tunnel seems to be flapping now. I can see it's up, but it's not receiving traffic sporatically. Tx seems to be normal, but after a little bit, the VPN tunnel closes, then reopens again and on goes the cycle.
I'm not seeing anything too odd in wireshark, however my assumption is the same, the connection seems to drop when I start to see alot of UDP connections from outside DNS address to the inside PC addresses as I posted an example above.
I'm going to continue watching traffic to see if I can find any other trends, but as it stands, the site is online using a SOHO router to the ISP.
After more investigating, looks like the ISP made some changes to their DHCP lease time which seems to be the cause. I assign a new IP it works for a day, then the lease expires. I'm reconfiguring and testing. i'll update when I have confirmaiton.
Experts Exchange Solution brought to you by
Your issues matter to us.
Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.
Start your 7-day free trialIt's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Networking
From novice to tech pro — start learning today.
Experts Exchange Solution brought to you by
Enjoy your complimentary solution view.
Get every solution instantly with Premium.
Start your 7-day free trial.