Strange Metro Ethernet Problem - Some IPs in the same subnet accessible, some not - Need Help
Posted on 2006-11-07
Quick run down of the setup here. I have a site that is using MetroEthernet for internet access provided by Time Warner (soon to be Comcast). Basically they have run a fiber connection to the building, it goes through a media converter, and plugs into our "border switch" via regular ethernet conneciton. Our firewall is plugged into this switch, and of course our corporate network is behind the firewall.
3 times now we have had a problem where some of our public IP addresses are accessible from the internet and some are not. I can plug a laptop into the "border switch" OUTSIDE of the firewall and get to ALL IP addresses and websites and services (which are 1to1 NATs thru the firewall) just fine. BUt anyone on the Internet can only access a few IPS (and each time we have the problem its different IP addresses)
Now this sounds like it would be an issue with the ISP, since we tested all CPE and it works fine. Here is the kicker. The first thing our ISP asks us to do is give our laptop (plugged into the "border switch", outside the firewall) one of the IP addresses that is inaccessible, we do, and he (and anyone on the internet) can get to. So from that one simple test he tells us this is a firewall problem (which i disagree, because I can access any services that are behind the firewall FROM outside the firewall, until I put the ISP network between the client and the firewall).
Now if we wait about 2-3 hours, everything starts working, TimeWarner claims they did nothing and I know we changed nothing.
Another issue that complicates this matter is that our default gateway (our first hop onto the internet) is a Layer3 switch (Catalyst 3550, so says the Time Warner guy) rather than a router, which worries me (even though I know they accomplish the same goal). It worries me because when I tracert to anything on the internet, my first HOP is NOT the same ip of my default gateway, and in my experience it is always best to have a router providing a solid first hop out. At the very least this helps troubleshooting.
Also 2 out of 3 times this has happened, we have been installing new firewall, and we get this strange behavoir. First time we thought it was the new firewall, but when we hooked the old one back up (which was working fine), same problem. My first instinct says this a MAC address/ ARP problem on the ISP end, and that traffic destined to certain IP addresses (all of which are NATed thru our firewall) is just not makin it to the right MAC address. But I dont know, that is why I am here.
Thanks in advance for the help!
I will be grateful to anyone who can provide assistance.