Solved

Strange Metro Ethernet Problem - Some IPs in the same subnet accessible, some not - Need Help

Posted on 2006-11-07
14
507 Views
Last Modified: 2008-02-01
Quick run down of the setup here.  I have a site that is using MetroEthernet for internet access provided by Time Warner (soon to be Comcast).  Basically they have run a fiber connection to the building, it goes through a media converter, and plugs into our "border switch" via regular ethernet conneciton. Our firewall is plugged into this switch, and of course our corporate network is behind the firewall.

3 times now we have had a problem where some of our public IP addresses are accessible from the internet and some are not.  I can plug a laptop into the "border switch" OUTSIDE of the firewall and get to ALL IP addresses and websites and services (which are 1to1 NATs thru the firewall) just fine.  BUt anyone on the Internet can only access a few IPS (and each time we have the problem its different IP addresses)

Now this sounds like it would be an issue with the ISP, since we tested all CPE and it works fine.  Here is the kicker.  The first thing our ISP asks us to do is give our laptop (plugged into the "border switch", outside the firewall) one of the IP addresses that is inaccessible, we do, and he (and anyone on the internet) can get to.  So from that one simple test he tells us this is a firewall problem (which i disagree, because I can access any services that are behind the firewall FROM outside the firewall, until I put the  ISP network between the client and the firewall).

Now if we wait about 2-3 hours, everything starts working, TimeWarner claims they did nothing and I know we changed nothing.

Another issue that complicates this matter is that our default gateway (our first hop onto the internet) is a Layer3 switch (Catalyst 3550, so says the Time Warner guy) rather than a router, which worries me (even though I know they accomplish the same goal).  It worries me because when I tracert to anything on the internet, my first HOP is NOT the same ip of my default gateway, and in my experience it is always best to have a router providing a solid first hop out.  At the very least this helps troubleshooting.

Also 2 out of 3 times this has happened, we have been installing new firewall, and we get this strange behavoir.  First time we thought it was the new firewall, but when we hooked the old one back up (which was working fine), same problem.  My first instinct says this a MAC address/ ARP problem on the ISP end, and that traffic destined to certain IP addresses (all of which are NATed thru our firewall) is just not makin it to the right MAC address.  But I dont know, that is why I am here.

Thanks in advance for the help!

I will be grateful to anyone who can provide assistance.
0
Comment
Question by:p0rkmonster
  • 6
  • 6
14 Comments
 
LVL 1

Expert Comment

by:BJHarris
ID: 17897237
It sounds to me like you need to troubleshoot this with the ISP until a conclusion is reached.  Which IPs are having the issue?  What type of firewall are you running?   When you isolate your network as far as possible, are you still having issues with the said IPs?  If so, it's Time Warner's issue.  If not, it's yours.
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17897272
The 3550 L3 switch is, for all intents and purposes, a router. There are many advantages to using this switch instead of what you would consider a real router.
What I would ask the ISP guy is whether or not I have a dedicated routed interface on that switch, or a VLAN interface. I agree that this appears to be an ARP issue especially since it magically goes away after a few hours- time for the cache to timeout. Is your default gateway IP address assiged to the physical interface of the 3550 that connects directly to you?  Given what you see in a traceroute this may not be the case, and it should be. Do you get a first hop timeout and then a reply from second hop? It could also simply be that the switch is answering as your first hop, just not with the router ID to match the IP address you would expect. This is perfectly normal.
I would also check for a duplex mismatch between your switch and their 3550 over the fiber transceiver. Look on your switch for port error counters.
0
 
LVL 1

Author Comment

by:p0rkmonster
ID: 17898236
BJHarris -

The firewall is a CyberGuard (ugh), but we get the same behavior with our new ASA 5510.  Each time we have the problem different IPs are affected.  From all our tests all Customer Premises Equipment works fine, up until the demarc. The reason why I need some sort of idea or "evidence" of a Time Warner problem, is because I am not directly responsible for this site (but I have servers there so I need it up) and the guy who is directly responsible is not  up to speed enough to do much troubleshooting or even understand it (dont ask, its corporate politics)so he has a hard time convicing Time Warner it is their fault (and so far I am too).

lrmoore -

My first hop out does not time out.  My default gateway ip is assinged to a VLAN interface, not physical (according to Time Warner tech).

I don't understand the relevance of a duplex mismatch?  Wouldn't that make all IP addresses have problems, not just a few?  Also I am no expert when it comes to the media converter, are they are any settings on it that may affect ARP?  I thought it was strictly a layer2 device?

Thanks for this and any future info.  I really appriciate it.
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17901296
Duplex mismatches cause all sorts of odd behavior and is certainly something to look into.
Media converters come in many "flavors" and not all support full-duplex. If it is provided by the ISP then I would expect that it would at least autonegotiate with your switch. It is strictly layer2 so can't affect ARP.
I'd be concerned with being assigned to a vlan interface and not a routed physical interface. If there are any other ports on the switch assigned to the same vlan then this could very easily be part of the problem. Anyone could use arp poisioning or any of a number of techniques to see what address range is supported by the connection and 'borrow' one of your IP's . . .
0
 
LVL 1

Author Comment

by:p0rkmonster
ID: 17915582
Thanks for the info so far guys.

I think not this weekend, but the next we may attempt to install our new firewall again, and see if this problem rears its ugly-head.

In the meantime I have been trying to find some official Cisco documention on best practices using 3550 as a router.  I would really like to be able to provide some type of Cisco gospel that encourages Time Warner to give us a real routed hop out to the internet, rather than this switched "ghost" hop, that does not show up on tracert, etc.  

Also why is it not a good idea to use a VLAN interface IP as the default gateway?  I dont like it, but from what I have read it is one solution for routing with this switch.  Please forgive my ignorance of this device.

Thanks.
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17915777
Cisco only provides the tools and I highly doubt that you could influence the ISP to change their business practice.
VLAN's are certainly one way to provide L3 services, but most often used only if you want more than one interface to be part of that L3 area/subnet/broadcast domain.

Consider an example config:

vlan 2
vlan 3
vlan 4

interface fast 0/2
 description customer ABC
 switchport access vlan2

interface fast 0/3
 description customer XYZ
 switchport access vlan3

interface fast 0/4
 description customer Acme
 switchport access vlan4

interface vlan 2
  ip address 12.34.56.1 255.255.255.248

interface vlan 3
  ip address 12.34.56.8 255.255.255.248

interface vlan 4
 ip address 12.34.56.16 255.255.255.248

With the above, all is well and good. Perhaps you are customer ABC and port Fast 0/2 has a connection to  your fiber transceiver. You get full use of all 5 remaining IP's in the subnet.
Now if a novice network engineer "accidently" assigns another switchport to your vlan, then this new customer is forced to use one or more of your IP addresses. If the 3550 switch is providing DHCP services to assign the customer equipment an IP address, then it will get one of yours.

interface fast 0/44
  switchport access vlan 2

OOPS, that should have been vlan 22.... easy mistake and detrimental to your network - and only to your network.

If, however, the switch is set up for 1 customer, 1 interface as a traditional router:
 interface Fast 0/2
  no switchport
  ip address 12.34.56.1 255.255.255.248
 interface Fast 0/2
  no switchport
  ip address 12.34.56.9 255.255.255.248
 interface Fast 0/3
  no switchport
  ip address 12.34.56.17 255.255.255.248
<etc>

Now there is no room for error. No possible way to assign any other customer to your VLAN.
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 1

Author Comment

by:p0rkmonster
ID: 17915923
I see your point, and you are probably correct about the ISP, he was insulted that I even suggested his setup was not optimal.


This leads me then to another question, something I have totally forgotten about.  Even though we are only using a single subnet of public IPs from the ISP, we actually pay for another seperate subnet (some old IPs that we didnt want to let go of).  Is it possible to assign more than one IP address to a VLAN interface?  If so what are the ramifications of this?  Since we have no traffic destined for any IPs on this old subnet, and we are not actually using any of the IPs on this old subnet, it would seem this is irrelevant, but again I am no expert when it comes to these new switches and I would be willing to bet that the ISP has configured this old subnet on the same switch/connection as the subnet we are currently using (since this is the only connection we have with the ISP).

0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17915982
> Is it possible to assign more than one IP address to a VLAN interface?
Sure. Just add a secondary IP. More traditionally that subnet would be simply routed to your existing IP on your Customer end.
 Following my example above, where you would have 12.34.56.2 on your CPE

EITHER:
 interface vlan2
  ip address 67.89.10.1 255.255.255.255 secondary

OR:
 ip route 67.89.10.0 255.255.255.0 12.34.56.2

Either way, the ISP has to broadcast their availability of that subnet to the rest of the world. If this subnet is not owned by this ISP, that won't happen.

0
 
LVL 1

Author Comment

by:p0rkmonster
ID: 17956518
Thanks everyone for the insight.

 I think we have the problem resolved.  It did turn out to be an ARP issue related to the Catalyst 3550.  Specifically the ISP did not keep the IOS on the switch up to date (just "a few" versions behind according to the ISP) and the version of IOS that was running had some bugs related to ARP/MAC address learning that was specific to the type of media converter (Radiance model made by a company called MetroOperability) being used by the ISP.

While the Cisco website does not detail the different devices the bug affected, the MetroOperability website did explain that the 3500 series of switches would cause problems if they were not running the current version of IOS.

Again thanks for the insight guys, yall provided me the ammunition necessary to force the ISP to admit the problem and fix this.  I am grateful.
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17959719
Thanks for the update. How do you want to close out this question?
0
 
LVL 1

Author Comment

by:p0rkmonster
ID: 17960169
well you deserve the points, If i accept one of your posts, will it still show everyone else the whole conversation?
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 17960705
Of course. The whole thread will then forever be available as a "solution" in the data base.
It is a valid problem/solution set, and since you did the research and found the real issue, we can get a moderator to accept your comment as the "answer" and put this away into PAQ status (Previously Answered Question) which is our way of saying close it out and put it in the data base...

Thanks!
0
 
LVL 1

Accepted Solution

by:
p0rkmonster earned 500 total points
ID: 17960959
Great!  I am new to the site (looked up stuff for years, just never posted or answered) so what is the procedure for getting this done as your described??
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Internet speed test 9 121
Internet redundancy 11 85
Draytek 2860 load balancing 6 23
Is Windows 10 taking more bandwidth than it ought to? 6 47
    Over the past few years, small business and home owners have become so dependent on internet that a need for redundancy has arisen.    What happens when your small business or home / home office loses its internet connection?  The results c…
Sometimes you have to pull out old tricks to get a new firewall to work… While we were installing a new Sonicwall at a customers site we found that sites they were able to visit before were not working.  It seemed random and we could not understa…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now