Solved

Static route to firewall drops intermittently.

Posted on 2009-05-11
5
482 Views
Last Modified: 2012-05-06
Hello all,

This is my first ever post to EE. Have found this resource an invaluable reference tool in the day to day life of an IT person. Thanks to all!

First, I am in no way a network engineer, or a Cisco savvy person. I know enough to get by.
Our setup up is simple ( I think ) We have an MPLS WAN setup with currently only two endpoints. Both endpoints have Cisco 2600 series routers provided by AT&T (a.k.a. the devil) Point A is the central office and will also host the internet connection. (AT&T MIS service) Point B (and every other future endpoint) will attach to services at the CO and get internet from the CO.

We do have a BGP number assigned to our WAN but for the whole WAN not every node on it. Right now being that we are basically using the MPLS as a PtP connection, I don't believe that  BGP is playing a part yet. BTW, AT&T has setup the config and is managing the routers for us, we have no access to them the routers themselves.

The problem:

 AT&T was to setup a static route on Point A router to point internet (outside traffic) to the IP of our firewall. But it didn't work. I had to set a static route statement on our firewall to get point B's
subnet to even see the IP of the firewall. Then Viola!  Point B is up on the internet, and can ping any node on Point A's LAN.

We then connected point A's LAN and Viola, we have internet access on both sides and can ping nodes on either side. BUT WAIT! all  the sudden, the route to the firewall just goes away out of the blue! We can still ping nodes on either sides LAN, but cant ping the IP of the firewall. But then just as sporadic, the internet comes back, and we can ping the firewall IP again. ????? Off and on like that all the time. We then disconnect the LAN of Point A and Point B is back to running internet through the WAN with no issue. The route to the firewall IP stays alive. I am baffled at this. We have ruled out hardware (at least our own) to blame as we have tried two different firewalls with similar configs with the same end results. We have spent hours on this issue ruling out everything that we can think of. Is the static route on the MPLS router to blame? Should that be there at all? If that route worked, would we still need the static route on our firewall? Are the two static routes conflicting? I am spit balling at this point.

Below is the topology and the running config from the Cisco 2600 router at point A's side. (had to basically give AT&T my right arm to get this)

We have NAT statements on the firewall, and access rules for servers on either side of the tunnel.


The topology:

Point A
-4.5Mb MLPPP connection to the MPLS cloud through Cisco 2600 router
-GE0/0 of router IP 192.168.1.2 (this is the address we are using as gateway on Point A's LAN)
-Point A subnet 192.168.1.0
-Internet Firewall IP 192.168.1.1 (Sonicwall Pro300)
-static route on firewall  dest:192.168.2.0 mask:255.255.255.0 GW:192.168.1.2
-static route on MPLS router (didnt work) 0.0.0.0 0.0.0.0 192.168.1.1

Point B
-3.0Mb MLPP connection to the MPLS cloud through Cisco 2600 router
-GE0/0 of router IP 192.168.2.2 (this is the address we are using as gateway on Point B's LAN)
-Point B subnet 192.168.2.0

Attachment: running config of POINT A MPLS router.

This is driving me insane, and AT&T is little help. just a round robin of call this number to find I have to then call another number to then get a ticket # to only find after two days it gets closed with no resolution. They keep checking the T1 circuits? Even though I never reported an issue about a hard down? ughh!

Any help is appreciated!

Brett


12-85-139-42-ACC-putty.log
0
Comment
Question by:peaveyone
  • 3
  • 2
5 Comments
 
LVL 7

Expert Comment

by:willbaclimon
ID: 24358836

Can you send a trace route when it works and when it doesnt? (you can mask the actual ip addresses for privacy)  Static routes take precedence are don't just drop out; which makes this problem easier to solve :)
0
 

Author Comment

by:peaveyone
ID: 24358896
@willbaclimon,

Thank you for your post,

No I cannot. when running trace route to 4.2.2.2 the sequence dies out at the hop just before my firewall IP of 192.168.1.1 which is the  serial IP of the MPLS router. However, when the route is back, (we can get to the internet) the traceroute runs fine. Then out of the blue will quit again. We tried two different Firewall devices thinking it was hardware. No joy.

Thanks,
Brett
0
 
LVL 7

Accepted Solution

by:
willbaclimon earned 500 total points
ID: 24358991

Sounds like a MPLS cloud issue. Usually the level 2-3 support guys are better at that.  Since AT&T manages the routers and you have static routes set up on both firewalls is 90% a carrier issue.  If you cant get anyone help, let me know. I have some contacts at at&t

0
 

Author Comment

by:peaveyone
ID: 24590535
The issue was indeed our LAN at one endpoint. At this endpoint, we have a firewall in a warehouse that sits in the LAN to divide a wifi network from the rest of the LAN. It turns out that the outside interface of that  firewall had the same IP address as the internet firewall's inside interface. However when pinging that IP address, we got no return, thus thinking the IP was unused. We found the problem in the ARP cache tables, the MAC would flip back and forth (before our eyes) when plugging and unplugging the internet firewall from the LAN. Strange issue that one was.

Brett
0
 

Author Closing Comment

by:peaveyone
ID: 31580151
Thank for your attempt to assist us! Much appreciated.
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Suggested Solutions

AWS has developed and created its highly available global infrastructure allowing users to deploy and manage their estates all across the world through the use of the following geographical components   RegionsAvailability ZonesEdge Locations  Wh…
Creating an OSPF network that automatically (dynamically) reroutes network traffic over other connections to prevent network downtime.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now