Static route to firewall drops intermittently.

Hello all,

This is my first ever post to EE. Have found this resource an invaluable reference tool in the day to day life of an IT person. Thanks to all!

First, I am in no way a network engineer, or a Cisco savvy person. I know enough to get by.
Our setup up is simple ( I think ) We have an MPLS WAN setup with currently only two endpoints. Both endpoints have Cisco 2600 series routers provided by AT&T (a.k.a. the devil) Point A is the central office and will also host the internet connection. (AT&T MIS service) Point B (and every other future endpoint) will attach to services at the CO and get internet from the CO.

We do have a BGP number assigned to our WAN but for the whole WAN not every node on it. Right now being that we are basically using the MPLS as a PtP connection, I don't believe that  BGP is playing a part yet. BTW, AT&T has setup the config and is managing the routers for us, we have no access to them the routers themselves.

The problem:

 AT&T was to setup a static route on Point A router to point internet (outside traffic) to the IP of our firewall. But it didn't work. I had to set a static route statement on our firewall to get point B's
subnet to even see the IP of the firewall. Then Viola!  Point B is up on the internet, and can ping any node on Point A's LAN.

We then connected point A's LAN and Viola, we have internet access on both sides and can ping nodes on either side. BUT WAIT! all  the sudden, the route to the firewall just goes away out of the blue! We can still ping nodes on either sides LAN, but cant ping the IP of the firewall. But then just as sporadic, the internet comes back, and we can ping the firewall IP again. ????? Off and on like that all the time. We then disconnect the LAN of Point A and Point B is back to running internet through the WAN with no issue. The route to the firewall IP stays alive. I am baffled at this. We have ruled out hardware (at least our own) to blame as we have tried two different firewalls with similar configs with the same end results. We have spent hours on this issue ruling out everything that we can think of. Is the static route on the MPLS router to blame? Should that be there at all? If that route worked, would we still need the static route on our firewall? Are the two static routes conflicting? I am spit balling at this point.

Below is the topology and the running config from the Cisco 2600 router at point A's side. (had to basically give AT&T my right arm to get this)

We have NAT statements on the firewall, and access rules for servers on either side of the tunnel.

The topology:

Point A
-4.5Mb MLPPP connection to the MPLS cloud through Cisco 2600 router
-GE0/0 of router IP (this is the address we are using as gateway on Point A's LAN)
-Point A subnet
-Internet Firewall IP (Sonicwall Pro300)
-static route on firewall  dest: mask: GW:
-static route on MPLS router (didnt work)

Point B
-3.0Mb MLPP connection to the MPLS cloud through Cisco 2600 router
-GE0/0 of router IP (this is the address we are using as gateway on Point B's LAN)
-Point B subnet

Attachment: running config of POINT A MPLS router.

This is driving me insane, and AT&T is little help. just a round robin of call this number to find I have to then call another number to then get a ticket # to only find after two days it gets closed with no resolution. They keep checking the T1 circuits? Even though I never reported an issue about a hard down? ughh!

Any help is appreciated!


Who is Participating?

Improve company productivity with a Business Account.Sign Up

willbaclimonConnect With a Mentor Commented:

Sounds like a MPLS cloud issue. Usually the level 2-3 support guys are better at that.  Since AT&T manages the routers and you have static routes set up on both firewalls is 90% a carrier issue.  If you cant get anyone help, let me know. I have some contacts at at&t


Can you send a trace route when it works and when it doesnt? (you can mask the actual ip addresses for privacy)  Static routes take precedence are don't just drop out; which makes this problem easier to solve :)
peaveyoneAuthor Commented:

Thank you for your post,

No I cannot. when running trace route to the sequence dies out at the hop just before my firewall IP of which is the  serial IP of the MPLS router. However, when the route is back, (we can get to the internet) the traceroute runs fine. Then out of the blue will quit again. We tried two different Firewall devices thinking it was hardware. No joy.

peaveyoneAuthor Commented:
The issue was indeed our LAN at one endpoint. At this endpoint, we have a firewall in a warehouse that sits in the LAN to divide a wifi network from the rest of the LAN. It turns out that the outside interface of that  firewall had the same IP address as the internet firewall's inside interface. However when pinging that IP address, we got no return, thus thinking the IP was unused. We found the problem in the ARP cache tables, the MAC would flip back and forth (before our eyes) when plugging and unplugging the internet firewall from the LAN. Strange issue that one was.

peaveyoneAuthor Commented:
Thank for your attempt to assist us! Much appreciated.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.