Link to home
Start Free TrialLog in
Avatar of thunderhead
thunderhead

asked on

Very strange ASA 5510 problem, dual isp / failover issue

Hi

We have a Cisco ASA 5510 set up with two inbound connections.
The config is set so that if the primary ISP cannot ping 4.2.2.2 it will failover to the secondary line, once the primary line can ping the same IP it will fail back.

Nothing out of the ordinary really for this bit of kit.

The set up is as follows:

Internet---> Primary ISP Equipment (x.x.x.249) --> Outside Primary on Cisco ASA (x.x.x.250)
Internet--> Seconadry ISP (x.x.x.30) --> Outside Secondary on Cisco ASA (x.x.x.30)

Cisco --> LAN

Earlier this month or ASA would not contact 4.2.2.2, it could ping the next hop address of the primary line (the ISPs equipment based in our office), naturally we phoned our ISP who, although slow to troubleshoot, said that the line was fine.

We confirmed this by attaching a laptop to the primary line

Internet---> Primary ISP Equipment (x.x.x.249) --> laptop (x.x.x.250)

And could ping 4.2.2.2, and browse the internet without problem.

As the Cisco was no longer covered by warranty / SmartNet we bought a new one.

After upgrading from 8.3(1) to 8.3.2 and finding that the Cisco would fail over, but would not fail back, we downgraded to 8.0(4), applied the config from the old router and were up and running.

Four days later the same problem emerged. We could ping the ISP equipment, but nothing beyond that. A laptop with a x.x.x.250 address would see the ISPs equipment (x.x.x.249) and anything beyond that, so the line was fine.

At the moment we are having to run with the following:

Internet---> Primary ISP Equipment (x.x.x.249) --> Netgear FVX538v2 (x.x.x.250)
Internet--> Seconadry ISP (x.x.x.30) --> Outside Secondary on Cisco ASA (x.x.x.30)

Then some funky routing to get things talking, but we are still having problems as servers cannot have two default gateways etc.

What we have tried so far:

Changing network cables (an obvious first step)
Confirming the Cisco config is ok - Both the ISP and the ASA supplier have confirmed that our config is absolutely fine
Confirming the line is ok - it works fine with a laptop, or with a netgear firewall.
Cisco hardware issue - it happened to one router, and then a brand new one? The first router had been in place without problem for two years.
Speed / Duplex issue? we have tried all combinations of 10/full, auto/auto, 100/full etc on the Cisco and nothing works
Placed a switch (unmanaged) between the primary line ISP equipment and the ASA to also eliminate the idea that it could be a port speed setting issue.
Turned the Cisco on and off, turned the port on and off.
Failover setting issue? Even if the failover / failback was not working between the two line we should still be able to ping out from the cisco primary line

An example of the ping responses from the Cisco (these are only a few seconds apart):

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to x.x.x.249, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:
!!!!?
Success rate is 80 percent (4/5), round-trip min/avg/max = 20/52/150 ms

 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)

We got a lot of responses like this, if we cleared the ARP cache on the Cisco we get a few ping responses initially then everything stops.

So far our supplier has not been able to come up with any ideas, neither have the ISP, and Cisco wont talk to us as we dont have a SmartNet contract with them.

Does any one else have any ideas what we could try, or have they seen anything like this in the past?
Avatar of Jimmy Larsson, CISSP, CEH
Jimmy Larsson, CISSP, CEH
Flag of Sweden image

Can you post your configuration here?

/Kvistofta
Avatar of thunderhead
thunderhead

ASKER

I made a boo-boo with the above - the set should read:

Internet---> Primary ISP Equipment (x.x.x.249) --> Outside Primary on Cisco ASA (x.x.x.250)
Internet--> Seconadry ISP (x.x.x.30) --> Outside Secondary on Cisco ASA (x.x.x.25)

I have attached the config, its been scrubbed of ip addresses etc.

Its been gone through with a fine tooth comb, by us, our ISP, and the asa suppliers but thanks for having a look!
cisco280910-scrubbed.txt
SOLUTION
Avatar of Jimmy Larsson, CISSP, CEH
Jimmy Larsson, CISSP, CEH
Flag of Sweden image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
may this simple tricks work;-

remove 4.2.2.2
add-8.8.8.8/8.8.4.4
@Kvistofta - it is static - but the backup line is a crappy BT router so normally its set to static with the .25 address, but at the moment its set to dynamic with the BT always giving it the same .25 address

@diprajbasu - that would show that problem might lie with the 4.2.2.2 address, but if we attach a laptop, or the netgear we can ping the 4.2.2.2 address with no problem I dont think its that, we have tried pinging the 8.8.4.4 address from the cisco using the primary line as the interface and it still cant get a response, or we get the same as above where it pings ok for say 80% then drops to 0 responses.
It looks strange. I wouldnt bet on it, but it could be worth a try to remove the static default route to the secondary isp. That shouldnt be needed since it will be inserted by the setroute anyway.

Can you show us output of "show route"?

/Kvistofta
Because the cisco isn't attached to the primary line at the moment (the netgear is) the sh route is as follows:

Gateway of last resort is x.x.x.30 to network 0.0.0.0

C    x.x.x.24 255.255.255.248 is directly connected, OUTSIDE_SECONDARY
S    172.20.0.0 255.255.0.0 [1/0] via 192.168.99.1, INSIDE
C    x.x.x.248 255.255.255.252 is directly connected, OUTSIDE_PRIMARY
S    10.2.0.0 255.255.0.0 [1/0] via x.x.x.30, OUTSIDE_PRIMARY
S    10.1.0.0 255.255.0.0 [1/0] via x.x.x.30, OUTSIDE_PRIMARY
d*   0.0.0.0 0.0.0.0 [1/0] via x.x.x.30, OUTSIDE_SECONDARY
C    192.168.0.0 255.255.0.0 is directly connected, INSIDE

The only thing odd I can see here is that the 2 vpns (10.1 and 10.2) show "via x.x.x.30, OUTSIDE_PRIMARY" (.30 is the BT line and is rthe secondary line).

The VPN config looks fine, and is set to use the primary line - I guess its because the primary is not accessible, so it will use the default.

I will try removing the secondary static route (its worth a shot), even though the config we had working for two years also had this in (it cant hurt to try!)
But are you in some kind of down-state now? Why is there no default route pointing to your primary ISP?

/Kvistofta
Because the cisco will not allow any traffic to go out on the primary line, so we have had to McGyver this up:

Internet---> Primary ISP Equipment (x.x.x.249) --> Netgear FVX538v2 (x.x.x.250)
Internet--> Secondary ISP (x.x.x.30) --> Outside Secondary on Cisco ASA (x.x.x.25)

This gets us close to normal working.

We are a 24 hour business so my window for troubleshooting is limited to about an hour and a half at 7am, and weekends.

I will be in again at the weekend, so if I can grab a "proper" routing table out of it (i.e. if the Cisco will allow traffic out long enough to realise that it has a route to the outside via the primary line), then I will post it.

*normally* there is a default route (to 0.0.0.0 0.0.0.0 via x.x.x.249, OUTSIDE_PRIMARY)
just for giggles, try a static route so that pings to 4.2.2.2 can only go out the primary interface

route OUTSIDE_PRIMARY 4.2.2.2 255.255.255.255 x.x.x.249
ASKER CERTIFIED SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial