Solved

10Giga Ethernet Link Between Two Buildings

Posted on 2013-12-23
46
828 Views
Last Modified: 2013-12-26
Hello Experts,

We have two buildings 1KM away from each other.

At Building 1, we have 2 6500 Switches in existence.

At Building 2, we are going to install the switches such as follows

Ground Floor - 1 x WS-C3750X-24S-E
                        2 x WS C3560X- 48S-P
First Floor        2x WS-C3560X-48T-L

Second Floor   2xWS-C3560X-48T-L

All these switches will be connected to Cisco Fiber Switch ( Ground Floor)

Fiber Switch will be connected to Main Data Center Fiber 6500 Switch.
These switches will be in separate VTP domain.

I need the help
- How to 10 Giga Speed between Fiber Switch and 6500
- Selection of Fiber Optic Cable which is installed through underground
-  Fiber Gbic

Moreover, I dont vlans from 6500 to get copy on Building 2 Switch.


Thanks in advance
0
Comment
Question by:cciedreamer
  • 22
  • 14
  • 10
46 Comments
 
LVL 22

Accepted Solution

by:
Jody Lemoine earned 350 total points
ID: 39736381
For a 1km link, you're looking at using single-mode G.652 fibre. If you were considering a 1GB connection, you could get away with multi-mode fibre with a 50-micron core, but you're not going to be able to do more than 400m with that at 10GB speeds.

The 3750X switch will need a C3KX-NM-10G module and a SFP-10G-LR SFP+ module. The Catalyst 6500 configuration will vary depending on what supervisors you're using. If you have supervisors that already support 10GB, then it's just a matter of getting an LR X2 or SFP+ module. If the supervisors don't support 10GB, then you'll either need to upgrade to supervisors that do or put in something like a WS-X6708-10G-3C module and a 10GBASE-LR X2 transceiver.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736395
Hello Sir,

Thanks for your response.

- How I can find out whether which supervisor is running on 6500
- WS-C3750X-24S-E has 24 fiber port, but I am not sure what is there speed.

Thanks
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736404
You can do a "show module" on the Catalyst 6500 to get the details on all installed modules, including the supervisors.

The 3750X's on-board ports are going to be 1Gb, regardless of whether it's a copper or fibre-based switch. That's why the C3KX-NM-10G module is necessary. It plugs into the expansion slot on the right-hand side of the unit and adds 10Gb SFP+ capability to the switch. Once that's installed, you can put an SFP-10G-LR SFP+ module into it in order to connect to your fibre.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736420
Just curious to know is will this be good lan design.

lan design
Result of Sh Module


sh module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL10425VXM
  2   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL10414EL3
  3   48  SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX     SAL1106GFFU
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       SAL1109JJMR

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  0019.2f57.1b74 to 0019.2f57.1b8b   2.5   12.2(14r)S5  12.2(18)SXD7 Ok
  2  0018.baaf.d9b8 to 0018.baaf.d9cf   2.5   12.2(14r)S5  12.2(18)SXD7 Ok
  3  001b.2a62.183c to 001b.2a62.186b  11.1   7.2(1)       8.3(0.156)RO Ok
  5  0016.46f9.2150 to 0016.46f9.2153   5.3   8.4(2)       12.2(18)SXD7 Ok

Mod Sub-Module                  Model              Serial        Hw     Status
--- --------------------------- ------------------ ------------ ------- -------
  1 Centralized Forwarding Card WS-F6700-CFC       SAL1107GYNF   2.1    Ok
  2 Centralized Forwarding Card WS-F6700-CFC       SAL1107GRJM   2.1    Ok
  5 Policy Feature Card 3       WS-F6K-PFC3B       SAL1102DRWS   2.3    Ok
  5 MSFC3 Daughterboard         WS-SUP720          SAL1109JM3P   2.6    Ok

Mod Online Diag Status
--- -------------------
  1 Pass
  2 Pass
  3 Pass
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736434
The Supervisor Engine 720 3B doesn't have the 10Gb uplinks, unfortunately. You'll need to either upgrade to the VS-S720-10G-3C (or VS-S720-10G-3CXL) or add a 10Gb module (mentioned previously) to have 10Gb connectivity to the switch. Adding the module is the less expensive option, but if you're considering a supervisor upgrade anyway then moving to one that supports the uplinks you need makes sense.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736448
Sir,
Just to confirm is this the part number

SFP-10G-LR SFP+

How about LAN design ? Is it fine performance and redundancy wise
- Any specs for fiber cable to be installed under the ground ?
0
 
LVL 22

Assisted Solution

by:Jody Lemoine
Jody Lemoine earned 350 total points
ID: 39736463
The part number is just SFP-10G-LR. The cable specifications were listed in my first comment: single-mode G.652 fibre.

Not enough information has been provided to actually do a design analysis, but the basic connectivity looks reasonable. You might want to consider getting a second 3750X and stacking it with the first one to eliminate a single point of failure in your building two network.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736472
Sir,

Just a conclusion for the up links connectivity

Cisco 6500   - Get only SFP-10G-LR module
Cisco 3750  -  Get only SFP-10G-LR module
Fiber Cable  - Single Mode G.652

Right Sir ?
0
 
LVL 22

Assisted Solution

by:Jody Lemoine
Jody Lemoine earned 350 total points
ID: 39736480
Nope.

6500: Get WS-X6708-10G-3C and X2-10GB-LR
3750: Get C3KX-NM-10G and SFP-10G-LR
Fibre Cable: Single Mode G.652
0
 
LVL 45

Assisted Solution

by:Craig Beck
Craig Beck earned 150 total points
ID: 39736485
Just as a note (in addition to the excellent advice jodylemoine has already given), G.652 fibre may be referred to as OS1.  It's the same thing, just a different name by a different governing body.  It may be better though to refer to it as OS1 as that differentiates it from OS2 (which annoyingly is also referred to as G.652).

With the 6500 10G connectivity, you may need to use a X2 transceiver instead of a SFP, as jodylemoine quite rightly said - it just depends on which 10G module you purchase for your 6500.  However if you want to use SFPs on all of your switches for consistency you could use the X2-CVR transceiver module to adapt the X2 slot to accommodate a SFP.  Just be aware that doing this adds something else to go wrong.  It's a perfectly acceptable solution though.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736492
Thank you Sir

As per the design is concerned.
- We have numerous vlans on B1 Core Switches,
- For Building 2 I want to create separate vlans for them and default to B1 Cores
- I don't want B1 vlans to pass through or copy to B2 Switch.
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39736503
So use VTP transparent on all your switches, and prune all VLANs which don't need to be trunked to each switch stack.

For example, you want VLANs 101-200 in B1, VLANs 201-300 in B2 and VLANs 301-400 in B3.

At the core, create all VLANs.

On the trunk link to B1 you do:

interface PortChannel1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport trunk allowed vlan 101-200
 switchport trunk native vlan 50


but on the trunk to B2 you do:

interface PortChannel2
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport trunk allowed vlan 201-300
 switchport trunk native vlan 50


...and so on.

Make sense?
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736504
If you want to be absolutely sure that this doesn't happen, you can set the 10Gb fibre interface on the 6500 as a routed interface. That way, no VLANs will pass at all. If you're going to do this, it makes more sense to configure the 3750X to be the default gateway for the building 2 VLANs and set the 10Gb fibre interfaces to be routed on both ends. Any traffic (if any) passing between the VLANs in building 2 can be handled locally by the 3750X and only traffic destined for the other building will be passed across the 10Gb fibre.
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736509
On the 3750X, you can do:

interface TenGigabitEthernet1/0/0
 no switchport
 ip address 10.10.10.1 255.255.255.252

On the 6500, you would do:

interface TenGigabitEthernet1/0
 no switchport
 ip address 10.10.10.2 255.255.255.252

(Interfaces and IPv4 addresses can be adjusted according to hardware configuration and addressing preference, of course.)

You can then define VLAN interfaces on the 3750X and put the default gateways for your VLANs there:

ip routing
!
interface VLAN50
 ip address 10.10.50.1 255.255.255.0
!
interface VLAN60
 ip address 10.10.60.1 255.255.255.0

The 3750X will be in control of your building 2 routing and will not leak VLANs to building 1, regardless of the VTP configuration on your switches in either building.

Then you can route between buildings using dynamic or static routing, and be absolutely sure that the VLANs in each of your buildings are separate.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736518
I read my mind. This is what I was thinking.

Can I also do in this way

B1 Core 1

interface vlan 50
ip address 10.10.50.1 255.255.255.0

interface Gi0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport trunk allowed vlan 101-200
 switchport trunk native vlan 50

but on the trunk to B2 you do:


interface vlan 50
ip address 10.10.50.2 255.255.255.0


interface gi0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport trunk allowed vlan 201-300
 switchport trunk native vlan 50
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736522
This will work too, but it requires that you take steps to isolate your VTP configurations in each building, either by making all switches transparent or by setting different domains and passwords. It will also mean that all inter-VLAN routing between VLANs in building 2 will traverse your fibre to building 1.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736531
I want to keep inter vlan routing for B2 the local switch itself

Can I achieve this with HSRP becoz we have this

On the 6500,

interface TenGigabitEthernet1/0
 no switchport
 ip address 10.10.10.2 255.255.255.252
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736537
If you want to keep inter-VLAN routing for B2 on the local switch itself, then the configuration needs to be on the 3750X and not on the 6500, so there's no need for HSRP there.

If you want inter-VLAN routing for B2 to be on the remote switch, then you would go with craigbeck's suggestion of passive VLANs across the 10Gb fibre and put your HSRP (if needed) on the VLAN interfaces on the 6500.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736560
I am thinking to configure this and keep inter vlan routing on local switch.

However I have redundancy at B1, If core 1 goes down, how B2 Switch will route the traffic to Core2 without any manual intervention.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736561
I am thinking to configure this and keep inter vlan routing on local switch.

However I have redundancy at B1, If core 1 goes down, how B2 Switch will route the traffic to Core2 without any manual intervention.
0
 
LVL 22

Assisted Solution

by:Jody Lemoine
Jody Lemoine earned 350 total points
ID: 39736574
If you have two 10Gb fibres running, one to each core, you can just run a subnet on each link and use a routing protocol to manage the connectivity:

Catalyst 3750X:

interface TenGigabitEthernet1/0/0
 no switchport
 ip address 10.10.10.2 255.255.255.252
!
interface TenGigabitEthernet1/0/1
 no switchport
 ip address 10.10.10.6 255.255.255.252
!
router eigrp 1010
 no auto-summary
 network 10.10.10.0 0.0.0.7
 eigrp stub connected

Catalyst 6500

interface TenGigabitEthernet1/0
 no switchport
 ip address 10.10.10.1 255.255.255.252
!
interface TenGigabitEthernet1/1
 no switchport
 ip address 10.10.10.5 255.255.255.252
!
router eigrp 1010
 no auto-summary
 network 10.10.10.0 0.0.0.7

Should either of the fibres go down, the routing protocol will route traffic over the other.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736648
I guess I should I  go with this Thanks

Just one 1 more query, I am going to place 15 U Wall Mounted Cabinet on each floor

Ground Floor Will hold 3 Switch

First floor will hold 2 x 48
Second floor will hold 2 x 48

Room Space Width 1m x Depth 1.5 m

You would like to shed some light, if thats fine.
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736661
None of those switches are more than a half metre from front to back, so that should be more than adequate.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 3

Author Comment

by:cciedreamer
ID: 39736723
Thank You Sir,

I will just make final plan if there is anything I'll update you.
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39736749
Agree with jodylemoine - L3 distribution layer is the way to go.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39736751
Thanks craigbeck
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39736771
You may be able to use both 10G links at the same time too if your 6500 switches are in a VSS.  That would be nice :-)
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39736778
Alas, he's using WS-SUP720-3B supervisors... so no VSS. I agree, that would have been nice.
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39736808
Ah yes - didn't notice that! I read 3C somewhere but now I see that was your post.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39737456
Hello Sir,

Which fiber cables is better single mode or double mode.

Thanks
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39737569
Single-mode will allow much more bandwidth over longer distances.

For example, with single-mode OS1 cable you can achieve 10G up-to 40km, but only 550m over multi-mode OM4 cable.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39737796
Sir,

Last thing I need to know whic LC connector I need for Switch 3750 10GB Speed.

Thank you experts,
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39737809
Not sure what you mean by "LC connector" here. The SFP+ needed for the single-mode connection was outlined earlier. Are you looking for patch cable information?
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39737816
Gbic Module i.e  GLC-LH-SM and along with 10 GB C3KX-NM-10G
0
 
LVL 22

Expert Comment

by:Jody Lemoine
ID: 39737822
Hey Samir. As we discussed earlier in this same topic, the 3750X needs a C3KX-NM-10G and one SFP-10G-LR for each fibre link connecting to it up to a maximum of two. The GLC-LH-SM is a GBIC and not appropriate for 10Gb connections at all.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39737830
Oopss...Sorry I didn't noticed that.

Thanks
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39737858
All SFPs use LC connectors.  The patch cord you need depends on what you're connecting the other end to.  If you're going direct from SFP to X2 module (for example) you'd need a LC-SC patch cord as the X2 module uses SC connectors.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39738503
Hi,
single-mode G.652 fibre can be installed underground

Or is there specific cable specs for them.

Thanks
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39738544
Yes.  The type of cable isn't really important, but the way in which the cable is protected is.  You can buy fibre cable which is protected by steel-wire, for example.  This will go in a duct between two buildings.
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39738585
Hi,

How about Metal Free Armoured Fiber Cable.
0
 
LVL 45

Expert Comment

by:Craig Beck
ID: 39738708
MFA is good... rats hate it!
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39738712
:)
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39739116
How many cores fiber cable should between buildings abd floors
0
 
LVL 45

Assisted Solution

by:Craig Beck
Craig Beck earned 150 total points
ID: 39739129
That depends on how many links you want to connect between buildings.  In your scenario a 4-core would do what you need.

I usually specify at least:

Total Cores = Required Cores + (Required Cores / 2)

So, if I need 2 'links' I'd usually require 4 cores (as each link needs 2 cores unless you use BX transceivers).  Using the formula that's 6 cores.  This will give you spare cores if you need them.

Obviously though if it's not going to cost much more you could specify say a 12-core fibre if you only need a 6-core.  It's always better to have more than you need.
0
 
LVL 3

Author Closing Comment

by:cciedreamer
ID: 39740260
craigbeck and  jodylemoine, I would like thank you for extensive support.

Thanks
0
 
LVL 3

Author Comment

by:cciedreamer
ID: 39740320
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Suggested Solutions

I eventually solved a perplexing problem setting up telnet for a new switch.  I installed a new Cisco WS-03560X-24P switch connected to an existing Cisco 4506 running a WS-X4013-10GE Sup II-Plus. After configuring vlans and trunking,  I could no…
Hello All, I have been training on Multicast for a while now and whenever I start the topic , I find out that my friends /  Colleagues mention that they do not know how to test Multicast Joins. As most of the multicast would be video traffic and …
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now