Solved

Weird Comcast routing

Posted on 2010-09-02
11
1,691 Views
Last Modified: 2013-12-14
Weird Comcast routing

I noticed that our latency from our Minneapolis office (which has 50/10 Comcast Business class) to a server in our Chicago office (a different ISP), is either routed a fast way through Chicago, or a slower route through Denver->Dallas->Chicago depending on which public, static IP address we connect from in Minneapolis.  Comcast assigned us 11.11.0.8/29 subnet, so there are 5 addresses I can assign after Comcast assigns our gateway 11.11.0.14.  We run a latency sensitive service over this connection.  We noticed it was either roughly 41 ms round trip, or 18 ms round trip.  Doing traceroutes from Minneapolis, we see it either routes through Denver, or through Chicago.  If we assign our Minneapolis server 11.11.0.9 or 11.11.0.11 it always routes through Chicago and is fast:


>ipconfig

Ethernet adapter Local Area Connection:

        Connection-specific DNS Suffix  . :
        IP Address. . . . . . . . . . . . : 11.11.0.11
        Subnet Mask . . . . . . . . . . . : 255.255.255.248
        Default Gateway . . . . . . . . . : 11.11.0.14

>tracert 66.28.82.19

Tracing route to 66.28.82.19 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  11-11-0-14-Minnesota.hfc.comcastbusiness.net [11.11.0.14]
  2     *        *        *     Request timed out.
  3     4 ms     2 ms     3 ms  ge-9-1-ur01.minnetonka.mn.minn.comcast.net [68.85.164.113]
  4    16 ms     7 ms     5 ms  te-8-1-ur02.crosstown.mn.minn.comcast.net [68.86.232.33]
  5     7 ms     7 ms     8 ms  68.87.174.193
  6    18 ms    17 ms    17 ms  te-0-4-0-5-cr01.chicago.il.ibone.comcast.net [68.86.91.185]
  7    17 ms    18 ms    17 ms  pos-0-0-0-0-pe01.350ecermak.il.ibone.comcast.net [68.86.86.34]
  8    18 ms    17 ms    21 ms  te0-2-0-1.ccr22.ord03.atlas.cogentco.com [154.54.11.253]
  9    17 ms    17 ms    17 ms  te0-0-0-1.ccr21.ord01.atlas.cogentco.com [154.54.1.1]
 10    21 ms    18 ms    17 ms  te0-1-0-0.mpd21.ord01.atlas.cogentco.com [154.54.30.250]
 11    19 ms   173 ms   294 ms  te4-2.mpd01.ord05.atlas.cogentco.com [154.54.3.206]
 12    19 ms    17 ms    18 ms  vl3805.na01.b000268-0.ord05.atlas.cogentco.com [66.28.6.86]
 13    18 ms    19 ms    17 ms  66.28.82.19

Trace complete.



If we assign the same server 11.11.0.10, 11.11.0.12, 11.11.0.13 It routes through the slower Denver route.

>ipconfig

Windows IP Configuration


Ethernet adapter Local Area Connection:

        Connection-specific DNS Suffix  . :
        IP Address. . . . . . . . . . . . : 11.11.0.12
        Subnet Mask . . . . . . . . . . . : 255.255.255.248
        Default Gateway . . . . . . . . . : 11.11.0.14

>tracert 66.28.82.19

Tracing route to 66.28.82.19 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  11-11-0-14-Minnesota.hfc.comcastbusiness.net [11.11.0.14]
  2     *        *        *     Request timed out.
  3     6 ms     7 ms     7 ms  ge-9-1-ur01.minnetonka.mn.minn.comcast.net [68.85.164.113]
  4     7 ms     9 ms     8 ms  te-8-1-ur02.crosstown.mn.minn.comcast.net [68.86.232.33]
  5     7 ms     9 ms     7 ms  te-8-3-ar02.roseville.mn.minn.comcast.net [68.87.174.6]
  6    26 ms    25 ms    25 ms  te-1-0-0-5-cr01.denver.co.ibone.comcast.net [68.86.93.133]
  7    43 ms    41 ms    41 ms  pos-0-8-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.85.178]
  8    43 ms    43 ms    46 ms  pos-0-1-0-0-pe01.1950stemmons.tx.ibone.comcast.net [68.86.86.94]
  9    41 ms    41 ms    41 ms  as174-pe01.1950stemmons.tx.ibone.comcast.net [75.149.230.150]
 10    42 ms    43 ms    42 ms  te7-4.mpd01.dfw01.atlas.cogentco.com [154.54.7.45]
 11    56 ms    53 ms    55 ms  te0-3-0-0.mpd22.mci01.atlas.cogentco.com [154.54.25.206]
 12    42 ms    46 ms    42 ms  te0-4-0-3.mpd22.ord01.atlas.cogentco.com [154.54.30.177]
 13    41 ms    41 ms    41 ms  te4-1.mpd01.ord05.atlas.cogentco.com [154.54.5.6]
 14    41 ms    41 ms    41 ms  vl3805.na01.b000268-0.ord05.atlas.cogentco.com [66.28.6.86]
 15    41 ms    41 ms    41 ms  66.28.82.19
 
Trace complete.
 
 
Is there a good reason for things to work this way?  Comcast tech support cannot answer.  I want the faster route for any of our public IP Address.
 
Note:  First hop IP address have been obfuscated here to protect the innocent.
0
Comment
Question by:pcrequest
  • 3
  • 3
  • 3
  • +1
11 Comments
 
LVL 16

Expert Comment

by:Bryan Butler
ID: 33587965
If you can test and you see that they are different routes and times for other places you test, then they have it setup (or it is setup by default) to route traffic different for your address ranges.  This is done on the comcast side.  If you ask them to look at the routing table entries for these address I think you will find they are different.  
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33588148

assuming the last octet .11 and .12 are the real Ip addresses
11.11.0.11
11.11.0.12

are in the same subnet

255.255.255.248

so the /29 should be more than likely in Comcast's IGP or even they have pushed all customer routes into BGP, but either case, traffic should go back the same path. I would ask Comcast to escalate the issue if you are a business customer; having sub-optimal routing for an IP in the same subnet is odd and is typically not normal.

Billy
0
 
LVL 16

Expert Comment

by:Bryan Butler
ID: 33588479
Billy,
Do you know if if the routing tables would show this?  Or if there is any other places in the router/swich configuration that would show this?
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33588818
can you post a traceroute from 66.28.82.19 to


11.11.0.11
11.11.0.12

See what you are seeing is the forward path, and this is up to the RIB/FIB, so what you are seeing could be possibly related to asymmetrical routing and/or equal cost paths; either case, Cogent does peer with Comcast in Chicago and in Dallas, so what you are seeing is sub-optimal routing (typically normal), but not normal in your case. Yes, this should be all in the routing table, but depending on the platform, like a Cisco that uses cef/dcef will be made by the FIB and not the RIB (There could be a mismatch between the RIB and FIB), either case, what you are seeing is NOT acceptable.

Billy
0
 
LVL 16

Accepted Solution

by:
Bryan Butler earned 167 total points
ID: 33589062
Sweet.  Thanks for the details Billy.  I was thinking there was another table to check.  So sub-optimal routing would be normal if they had facilities in different cities, or a special case such as you specifically setup the office so HR could connect to Chicago where the HR DB was, and others would go through Dallas by default, eh?  And it is not normal in his case because they probably haven't asked for this, and they probably should go the same route if all the nodes/ip-addresses are in the same location, eh?
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 24

Assisted Solution

by:rfc1180
rfc1180 earned 167 total points
ID: 33589249
>And it is not normal in his case because they probably haven't asked for this
it is nothing you can ask for, it is all Dependant on where the user is located, whom their ISP peers with.
Typically, there is nothing you can do about suboptimal routing if 2 ISPs do not BGP peer with each other, however, sub-optimal routing in a single autonomous system is typically now normal, and this is what you are seeing (Sub-optimal routing in Comcasts network); however, I am not going to pretend to know how and what Comcast is doing in their network (This might be expected to them).

I would open a ticket/escalate and advise why is it that you traverse all the way to Dallas just to get to Chicago from MN. What you have provided, again, in my professional opinion is not normal.

Billy  
0
 
LVL 1

Author Comment

by:pcrequest
ID: 33599759
http://traceroute.org find "comcast".  Anything to be gleaned from that tool?  It launches my hyperterm, I can log in and fart around with some traceroutes and such, but I don't know anything about BGP...

By the way, yes, the last octets are real in my examples.  All in the same network ID.  All from the same building.  All we asked for was a /29 block of IPs (no special routing).  I tracert'd some other targets, and notably, routes are consistent within a specific source IP address. Route doesn't always make sense from source IP to source IP.  From 11.11.0.9 may take a direct route to our Chicago server, but a Denver route to our mail server in Vancouver (look at a Canadian map :), but those routes never change for 11.11.0.9. To clarify, it doesn't change for any of our source IPs. 11.11.0.10 is Denver to Chicago server destination(bad), but takes Chicago route to Vancouver mail server (good).

I did routes back from our Chicago server. Traffic routes back through Chicago.  Note .11, .12, and .13 are the same server.  The others are various routers.

I'd like a little more ammo before I escalate this with Comcast.
>tracert 11.11.0.9

Tracing route to 11-11-0-9-Minnesota.hfc.comcastbusiness.net [11.11.0.9]
over a maximum of 30 hops:

  1    11 ms     9 ms     9 ms  10.10.10.3
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4    <1 ms    <1 ms    <1 ms  66.28.82.19
  5     1 ms     1 ms     1 ms  fa0-17.na01.b000268-0.ord05.atlas.cogentco.com [66.28.28.17]
  6    59 ms     3 ms    18 ms  gi1-40.3805.mpd01.ord05.atlas.cogentco.com [66.28.6.85]
  7     1 ms     1 ms     1 ms  te0-4-0-6.mpd22.ord01.atlas.cogentco.com [154.54.5.5]
  8     1 ms     1 ms     1 ms  te0-1-0-0.ccr22.ord01.atlas.cogentco.com [154.54.6.178]
  9     5 ms     1 ms     1 ms  te0-1-0-7.ccr22.ord03.atlas.cogentco.com [154.54.1.98]
 10     1 ms     1 ms     1 ms  comcast.ord03.atlas.cogentco.com [154.54.11.254]
 11     1 ms     1 ms     1 ms  pos-1-4-0-0-cr01.chicago.il.ibone.comcast.net [68.86.86.157]
 12    12 ms    11 ms    11 ms  te-0-2-0-7-ar01.roseville.mn.minn.comcast.net [68.86.91.6]
 13    13 ms    12 ms    12 ms  te-0-2-0-0-ar01.crosstown.mn.minn.comcast.net [68.87.174.210]
 14    35 ms    35 ms    35 ms  te-2-1-ur01.martin.mn.minn.comcast.net [68.86.232.34]
 15    35 ms    35 ms    35 ms  68.85.164.114
 16    16 ms    17 ms    24 ms  73.237.189.113
 17     *        *        *     Request timed out.


>tracert 11.11.0.10

Tracing route to 11-11-0-10-Minnesota.hfc.comcastbusiness.net [11.11.0.10]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  10.10.10.3
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4    <1 ms    <1 ms    <1 ms  66.28.82.19
  5     1 ms     1 ms     1 ms  fa0-17.na01.b000268-0.ord05.atlas.cogentco.com [66.28.28.17]
  6   139 ms    27 ms     4 ms  gi1-40.3805.mpd01.ord05.atlas.cogentco.com [66.28.6.85]
  7     1 ms     1 ms     1 ms  te0-4-0-6.mpd21.ord01.atlas.cogentco.com [154.54.3.205]
  8     1 ms     1 ms    <1 ms  te0-1-0-0.ccr21.ord01.atlas.cogentco.com [154.54.30.249]
  9     2 ms     1 ms     1 ms  te0-1-0-3.ccr22.ord03.atlas.cogentco.com [154.54.1.2]
 10     4 ms     1 ms     1 ms  comcast.ord03.atlas.cogentco.com [154.54.10.254]
 11     1 ms     1 ms     1 ms  pos-1-4-0-0-cr01.chicago.il.ibone.comcast.net [68.86.86.157]
 12    11 ms    11 ms    11 ms  te-0-3-0-5-ar01.roseville.mn.minn.comcast.net [68.86.91.186]
 13    12 ms    12 ms    12 ms  te-0-2-0-0-ar01.crosstown.mn.minn.comcast.net [68.87.174.210]
 14    35 ms    35 ms    35 ms  te-2-1-ur01.martin.mn.minn.comcast.net [68.86.232.34]
 15    12 ms    12 ms    12 ms  68.85.164.122
 16    16 ms    17 ms    17 ms  73.237.189.113
 17    40 ms    40 ms    39 ms  11-11-0-10-Minnesota.hfc.comcastbusiness.net [11.11.0.10]

Trace complete.


>tracert 11.11.0.11

Tracing route to 11-11-0-11-Minnesota.hfc.comcastbusiness.net [11.11.0.11]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  10.10.10.3
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4    <1 ms    <1 ms    <1 ms  66.28.82.19
  5     1 ms     1 ms     1 ms  fa0-17.na01.b000268-0.ord05.atlas.cogentco.com [66.28.28.17]
  6    12 ms     5 ms   264 ms  gi1-48.3505.mpd01.ord05.atlas.cogentco.com [66.28.6.81]
  7     1 ms    <1 ms     1 ms  te0-4-0-6.mpd22.ord01.atlas.cogentco.com [154.54.5.5]
  8     1 ms     1 ms     1 ms  te0-1-0-7.ccr21.ord03.atlas.cogentco.com [154.54.41.58]
  9     1 ms     1 ms     1 ms  te0-1-0-0.ccr22.ord03.atlas.cogentco.com [66.28.4.66]
 10     1 ms     1 ms     1 ms  comcast.ord03.atlas.cogentco.com [154.54.10.254]
 11     2 ms     2 ms     1 ms  pos-1-4-0-0-cr01.chicago.il.ibone.comcast.net [68.86.86.157]
 12    11 ms    11 ms    11 ms  te-0-3-0-5-ar01.roseville.mn.minn.comcast.net [68.86.91.186]
 13    13 ms    12 ms    12 ms  68.87.174.194
 14    35 ms    35 ms    35 ms  te-2-1-ur01.martin.mn.minn.comcast.net [68.86.232.34]
 15    35 ms    35 ms    35 ms  68.85.164.114
 16    17 ms    17 ms    15 ms  73.237.189.113
 17    40 ms    39 ms    40 ms  11-11-0-11-Minnesota.hfc.comcastbusiness.net [11.11.0.11]

Trace complete.


>tracert 11.11.0.12

Tracing route to 11-11-0-12-Minnesota.hfc.comcastbusiness.net [11.11.0.12]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  10.10.10.3
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4    <1 ms    <1 ms    <1 ms  66.28.82.19
  5     1 ms     1 ms     1 ms  fa0-17.na01.b000268-0.ord05.atlas.cogentco.com [66.28.28.17]
  6     1 ms     1 ms     1 ms  gi1-40.3805.mpd01.ord05.atlas.cogentco.com [66.28.6.85]
  7     1 ms     1 ms     1 ms  te0-4-0-6.mpd22.ord01.atlas.cogentco.com [154.54.5.5]
  8     1 ms    <1 ms     1 ms  te0-1-0-0.ccr22.ord01.atlas.cogentco.com [154.54.6.178]
  9     1 ms     1 ms     1 ms  te0-1-0-7.ccr22.ord03.atlas.cogentco.com [154.54.1.98]
 10     1 ms     1 ms     1 ms  comcast.ord03.atlas.cogentco.com [154.54.11.254]
 11     1 ms     2 ms     1 ms  pos-1-6-0-0-cr01.chicago.il.ibone.comcast.net [68.86.86.77]
 12    11 ms    11 ms    11 ms  68.86.95.6
 13    13 ms    12 ms    12 ms  te-0-1-0-2-ar02.crosstown.mn.minn.comcast.net [68.87.174.225]
 14    35 ms    38 ms    35 ms  te-2-1-ur01.martin.mn.minn.comcast.net [68.86.232.34]
 15    35 ms    35 ms    35 ms  68.85.164.114
 16    21 ms    19 ms    19 ms  73.237.189.113
 17    20 ms    19 ms    23 ms  11-11-0-12-Minnesota.hfc.comcastbusiness.net [11.11.0.12]

Trace complete.



>tracert 11.11.0.13

Tracing route to 11-11-0-13-Minnesota.hfc.comcastbusiness.net [11.11.0.13]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  10.10.10.3
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4    <1 ms    <1 ms    <1 ms  66.28.82.19
  5     1 ms     1 ms     1 ms  fa0-17.na01.b000268-0.ord05.atlas.cogentco.com [66.28.28.17]
  6     2 ms     3 ms     1 ms  gi1-48.3505.mpd01.ord05.atlas.cogentco.com [66.28.6.81]
  7     1 ms     1 ms     1 ms  te0-4-0-6.mpd21.ord01.atlas.cogentco.com [154.54.3.205]
  8     3 ms     1 ms     1 ms  te0-1-0-3.ccr21.ord03.atlas.cogentco.com [154.54.41.54]
  9     1 ms     1 ms     1 ms  te0-0-0-0.ccr22.ord03.atlas.cogentco.com [154.54.24.214]
 10     1 ms     1 ms     1 ms  comcast.ord03.atlas.cogentco.com [154.54.11.254]
 11     2 ms     2 ms     1 ms  pos-1-5-0-0-cr01.chicago.il.ibone.comcast.net [68.86.86.161]
 12    11 ms    11 ms    11 ms  te-0-4-0-2-ar01.roseville.mn.minn.comcast.net [68.86.91.138]
 13    13 ms    12 ms    13 ms  68.87.174.194
 14    35 ms    35 ms    36 ms  te-2-1-ur01.martin.mn.minn.comcast.net [68.86.232.34]
 15    13 ms    12 ms    12 ms  68.85.164.122
 16    18 ms    19 ms    15 ms  73.237.189.113
 17    22 ms    17 ms    20 ms  11-11-0-13-Minnesota.hfc.comcastbusiness.net [11.11.0.13]

 
 
 >tracert 11.11.0.14

Tracing route to 11-11-0-14-Minnesota.hfc.comcastbusiness.net [11.11.0.14]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  10.10.10.3
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4    <1 ms    <1 ms    <1 ms  66.28.82.19
  5     1 ms     1 ms     1 ms  fa0-17.na01.b000268-0.ord05.atlas.cogentco.com [66.28.28.17]
  6   746 ms   348 ms     2 ms  gi1-48.3505.mpd01.ord05.atlas.cogentco.com [66.28.6.81]
  7     1 ms     1 ms     1 ms  te0-4-0-6.mpd22.ord01.atlas.cogentco.com [154.54.5.5]
  8     1 ms     1 ms     1 ms  te0-1-0-7.ccr21.ord03.atlas.cogentco.com [154.54.41.58]
  9     2 ms     1 ms     1 ms  te0-1-0-0.ccr22.ord03.atlas.cogentco.com [66.28.4.66]
 10     2 ms     2 ms     1 ms  comcast.ord03.atlas.cogentco.com [154.54.10.254]
 11     2 ms     2 ms     2 ms  pos-1-6-0-0-cr01.chicago.il.ibone.comcast.net [68.86.86.77]
 12    11 ms    12 ms    11 ms  te-0-2-0-7-ar01.roseville.mn.minn.comcast.net [68.86.91.6]
 13    12 ms    12 ms    13 ms  te-0-1-0-2-ar02.crosstown.mn.minn.comcast.net [68.87.174.225]
 14    35 ms    36 ms    35 ms  te-2-1-ur01.martin.mn.minn.comcast.net [68.86.232.34]
 15    35 ms    35 ms    35 ms  68.85.164.114
 16    43 ms    41 ms    40 ms  11-11-0-14-Minnesota.hfc.comcastbusiness.net [11.11.0.14]

Trace complete.

Open in new window

0
 
LVL 3

Assisted Solution

by:cwoolsey
cwoolsey earned 166 total points
ID: 33991652
I guess the concern I see here is a lot of discussion about the routing but the major question is there a special configuration for this data to be on a private connection or some sort of MPLS or QOS based link between the sites? If you do not have that as part of your services you are basically wasting your time. All ISP's and carriers charge for private network routing and management. Often configured as an MPLS network connection. Some providers can do this as a "layered" service on an existing circuit others have to bring in seperate circuits to handle this. Some providers do not offer this at all. All carriers use least cost routing for the most part and most cable providers are not big on changing routes for a customer unless they have added in the private routing services.

I did not see anything above that lists that you have paid for those types of services so they can route you to california and back and there is really nothing they will do for you as it is not a service you have paid for. you can have 4 locations all with comcast as the provider but they may route you out to the public internet at points simply because you are not paying them to keep you on thier private network.  network backhauls cost carrier's money and those costs are passed onto the customer's who require and pay for those types of services. Otherwise you are at the mercy of possible dynamic least cost routing and they may have a tolerable latency policy but it can be extremely diverse based on network loads, time of day, etc.

If you are having issues you really need to look at a managed MPLS network with guaranteed QOS services. Comcast may offer this or they may not I am not completely sure. But most Comcast networks are also shared networks so therefore your bandwidth can vary depending on the customer density in your area as well. Again another reason to look at managed MPLS networks. these often use T1's, Bonded T1's, Ethernet over copper, or Fiber for the last mile.
0
 
LVL 1

Author Comment

by:pcrequest
ID: 34026483
cwoolsey, what's odd is my test server, depending only on which of the /29 addresses I gave it, would determine what route it took.  If they are doing least cost routing, it's coming from the same server, I'd expect the same route.

Yes, it is a VPN and we are at the mercy of however Comcast wants to route.

What's interesting, and my apologies for not keep this up-to-date, is about a week after I complained to Comcast and they basically said "it is what it is", it mysteriously got fixed and has stayed (from my perspective that way since.

I found www.routeplotter.com software helpful to keep tabs on route changes.  Anyone know of similar, free software to keep track of tracerts over time?

Thanks all for the interesting discussions.  I will review and assign points soon based on the value of your comments.
0
 
LVL 1

Author Closing Comment

by:pcrequest
ID: 34455554
Thanks for the discussion.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Configuring network clients can be a chore, especially if there are a large number of them or a lot of itinerant users.  DHCP dynamically manages this process, much to the relief of users and administrators alike!
Shadow IT is coming out of the shadows as more businesses are choosing cloud-based applications. It is now a multi-cloud world for most organizations. Simultaneously, most businesses have yet to consolidate with one cloud provider or define an offic…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now