?
Solved

VLAN problem with remote site on Metro Ethernet

Posted on 2008-11-10
18
Medium Priority
?
4,784 Views
Last Modified: 2012-05-05
Hello EE,

Many of you have been following my posts pertaining to a major upgrade to Metro Ethernet.  Last Thursday evening, we moved forward with 4 of our sites.  Everything went very smoothly until we got to SITE D.  This is when we ran across what we believe to be a VLAN tag related issue.  Here is the rundown of what we did.

SITE A:
We installed a Cisco 2821 router in our main data center.  This new router is touching the metro cloud and we assigned the metro port the address 10.10.10.1.  Another port on this device is connected back to our existing network infrastructure.  Remote sites are to be migrated from the existing network to the fiber one at a time.  All routes are good and tested.  The Router is up and routing traffic right now.

SITE B:
We installed a Cisco 2811 router in the data center at our Police Station.  This router is touching the metro cloud and has an address of 10.10.10.2.  The other port on this device is connected to the switches that contain all the users and server on the 192.168.111.0 subnet.  Addresses are statically assigned.  All routes are good and tested.  The Router is up and routing traffic right now.

SITE C:
We installed a Cisco 3560 switch on this site.  L2 routing capabilities are enabled.  The switch is touching the metro cloud and has an address of 10.10.10.3.  There is a VLAN20 interface with the address 192.168.109.1.  All the users connected to this switch are assigned to VLAN20, and address are assigned through DHCP.  All routes are good and tested.

SITE D:
We installed a Cisco 3560 switch on this site.  The switch and the configuration is identical to the switch at SITE C, with the exception of the hostname and the IP addresses.  The switch is touching the metro cloud and has an address of 10.10.10.4.

Here is where the problem begins.   When I plugged it in and did a few housekeeping procedures (i.e. removed erroneous routes) interface VLAN20 would not come up no matter what I tried.  From the switch console i could ping everything on the network, but nobody else could see past the metro port at site D.  Here is some of the the step I took to resolve the issue.

blew away the vlan20 interface and started over - NO CHANGE
created a new vlan interface (VLAN220) and assigned the network ip address to that - NO CHANGE
reloaded the config from our TFTP server - NO CHANGE
reloaded the config from the switch at SITE C and changed hostname, ip's, etc - NO CHANGE
replaced with a whole different switch - NO CHANGE

Nothing I did would bring that vlan interface up.  Finally our desperation and pure exhaustion, at 11:45, I decided to assigned the network ip to the vlan1 interface and what do you know, all the ports on the switch that were lit up amber all turned green and all the pc's at SITE D started grabbing DHCP.  We desided to leave it alone for now and research what went wrong.  so my question is this.

Why would the VLAN20 interface not come online?  
Did it have something to do with an active VLAN20 running at site C?  

An earlier post here suggests that the switches are oblivious to the VLANS on other switches in a setup like this.  

What more should I be looking at?  
What are the dangers of running traffic on VLAN1?  

The hard part is done.  Now we need to work out the kinks before we move the other 7 sites over.  I have attached a diagram and the configs I want to use for reference.  
~~~~~~~~~~~~~~~~~~~~~SITE A~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname CHR1
!
boot-start-marker
boot system flash:c2800nm-adventerprisek9-mz.124-20.T1.bin
boot-end-marker
!
logging message-counter syslog
logging buffered 51200 warnings
enable secret 5 *****
enable password 1*****
!
no aaa new-model
!
dot11 syslog
ip source-route
!
!
no ip cef
!
!
no ipv6 cef
!
multilink bundle-name authenticated
!
!
voice-card 0
 no dspfarm
!
!
username netmaster privilege 15 secret 5 *****
archive
 log config
  hidekeys
!
!
interface GigabitEthernet0/0
 description VLAN30 SERVERS
 shutdown
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description METRO ETHERNET
 ip address 10.10.10.1 255.255.255.240
 ip helper-address 192.168.101.215
 duplex auto
 speed auto
!
interface FastEthernet0/0/0
 description VLAN10 MGMT-IT
 switchport access vlan 10
!
interface FastEthernet0/0/1
 description ASA 5510 FIREWALL
!
interface FastEthernet0/0/2
 description VLAN20 CITY HALL
 switchport access vlan 20
!
interface Serial0/1/0
 description T1 DIRECT LINK AIRPORT
 ip address 192.168.1.25 255.255.255.248
!
interface FastEthernet0/2/0
 description LINK TO OLD NETWORK
 ip address 192.168.101.5 255.255.255.0
 duplex auto
 speed auto
!
interface Vlan1
 no ip address
!
interface Vlan10
 description MGMT DEVICES CONNECTED TO FE0/0/0
 ip address 192.168.96.1 255.255.255.0
!
interface Vlan20
 description CITY HALL DEVICES CONNECTED TO FE0/0/2
 ip address 192.168.100.1 255.255.255.0
!
router eigrp 1
 network 192.168.96.0
 network 192.168.100.0
 network 192.168.101.0
 network 192.168.1.0
 network 10.10.10.0
 auto-summary
!
ip classless
ip route 0.0.0.0 0.0.0.0 FastEthernet0/2/0
ip route 192.168.96.0 255.255.255.0 FastEthernet0/0/0
ip route 192.168.100.0 255.255.255.0 FastEthernet0/0/2
ip route 192.168.101.0 255.255.255.0 GigabitEthernet0/0
ip route 192.168.102.0 255.255.255.0 192.168.101.9
ip route 192.168.103.0 255.255.255.0 192.168.101.9
ip route 192.168.114.0 255.255.255.0 192.168.101.9
ip route 192.168.104.0 255.255.255.0 10.10.10.5
ip route 192.168.105.0 255.255.255.0 10.10.10.6
ip route 192.168.106.0 255.255.255.0 10.10.10.7
ip route 192.168.107.0 255.255.255.0 10.10.10.9
ip route 192.168.108.0 255.255.255.0 Serial0/1/0
ip route 192.168.109.0 255.255.255.0 10.10.10.3
ip route 192.168.110.0 255.255.255.0 10.10.10.4
ip route 192.168.111.0 255.255.255.0 10.10.10.2
ip route 192.168.112.0 255.255.255.0 10.10.10.10
ip route 192.168.113.0 255.255.255.0 10.10.10.8
no ip http server
no ip http secure-server
!
!
control-plane
!
!
line con 0
 login local
line aux 0
line vty 0 4
 access-class 23 in
 privilege level 15
 password *****
 login local
 transport input telnet
line vty 5 15
 access-class 23 in
 privilege level 15
 password *****
 login local
 transport input telnet
!
scheduler allocate 20000 1000
end
 
~~~~~~~~~~~~~~~~~~~~~SITE B~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname PDR1
!
boot-start-marker
boot-end-marker
!
enable secret 5 *****
enable password *****
!
no aaa new-model
!
resource policy
!
ip subnet-zero
!
!
no ip cef
!
!
!
!
!
interface FastEthernet0/0
 description VLAN100 traffic from CHR1
 ip address 10.10.10.2 255.255.255.240
 duplex auto
 speed auto
!
interface FastEthernet0/1
 description KPD SWITCH
 ip address 192.168.111.1 255.255.255.0
 duplex half
 speed auto
 no mop enabled
!
router eigrp 1
 network 10.10.10.0
 network 192.168.111.0
 auto-summary
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.10.10.1
ip route 192.168.96.0 255.255.255.0 10.10.10.1
ip route 192.168.100.0 255.255.255.0 10.10.10.1
ip route 192.168.101.0 255.255.255.0 10.10.10.1
ip route 192.168.102.0 255.255.255.0 10.10.10.1
ip route 192.168.103.0 255.255.255.0 10.10.10.1
ip route 192.168.114.0 255.255.255.0 10.10.10.1
ip route 192.168.104.0 255.255.255.0 10.10.10.5
ip route 192.168.105.0 255.255.255.0 10.10.10.6
ip route 192.168.106.0 255.255.255.0 10.10.10.7
ip route 192.168.107.0 255.255.255.0 10.10.10.9
ip route 192.168.108.0 255.255.255.0 10.10.10.1
ip route 192.168.109.0 255.255.255.0 10.10.10.3
ip route 192.168.110.0 255.255.255.0 10.10.10.4
ip route 192.168.111.0 255.255.255.0 FastEthernet0/1
ip route 192.168.112.0 255.255.255.0 10.10.10.10
ip route 192.168.113.0 255.255.255.0 10.10.10.8
no ip http server
no ip http secure-server
!
!
control-plane
!
!
line con 0
line aux 0
line vty 0 4
 password *****
 login
!
scheduler allocate 20000 1000
!
end
 
~~~~~~~~~~~~~~~~~~~~~SITE C~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname COURT
!
enable secret 5 *****
enable password *****
!
no aaa new-model
system mtu routing 1500
ip subnet-zero
ip routing 
!
!
!
!
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
interface FastEthernet0/1
 switchport mode access
 switchport access vlan 20
 spanning-tree portfast
!
~~~~~~~~INTERFACES TRUNCATED
!
interface FastEthernet0/24
 description VLAN20 traffic from CHR1
 no switchport
 ip address 10.10.10.3 255.255.255.240
!
interface GigabitEthernet0/1
!
interface GigabitEthernet0/2
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan10
 description MGMT ACCESS
 ip address 192.168.96.51 255.255.255.0
!
interface Vlan20
 description COURT
 ip address 192.168.109.1 255.255.255.224
 ip helper-address 192.168.101.215
!
router eigrp 1
 network 10.10.10.0
 network 192.168.96.0
 network 192.168.109.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.10.10.1
ip route 192.168.96.0 255.255.255.0 10.10.10.1
ip route 192.168.100.0 255.255.255.0 10.10.10.1
ip route 192.168.101.0 255.255.255.0 10.10.10.1
ip route 192.168.102.0 255.255.255.0 10.10.10.1
ip route 192.168.103.0 255.255.255.0 10.10.10.1
ip route 192.168.114.0 255.255.255.0 10.10.10.1
ip route 192.168.104.0 255.255.255.0 10.10.10.5
ip route 192.168.105.0 255.255.255.0 10.10.10.6
ip route 192.168.106.0 255.255.255.0 10.10.10.7
ip route 192.168.107.0 255.255.255.0 10.10.10.9
ip route 192.168.108.0 255.255.255.0 10.10.10.1
ip route 192.168.110.0 255.255.255.0 10.10.10.4
ip route 192.168.111.0 255.255.255.0 10.10.10.2
ip route 192.168.112.0 255.255.255.0 10.10.10.10
ip route 192.168.113.0 255.255.255.0 10.10.10.8
no ip http server
!
!
control-plane
!
!
line con 0
line vty 0 4
 password *****
 login
line vty 5 15
 password *****
 login
!
end
 
~~~~~~~~~~~~~~~~~~~~~SITE D~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname KSP
!
enable secret 5 *****
enable password *****
!
no aaa new-model
system mtu routing 1500
ip subnet-zero
ip routing
!
!
!
!
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
interface FastEthernet0/1
 switchport mode access
 switchport access vlan 20
 spanning-tree portfast
!
~~~~~~~~~~~~~INTERFACES TRUNCATED
!
interface FastEthernet0/24
 description METRO ETHERNET PORT
 no switchport
 ip address 10.10.10.4 255.255.255.240
!
interface GigabitEthernet0/1
!
interface GigabitEthernet0/2
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan10
 description MGMT ACCESS
 ip address 192.168.96.54 255.255.255.0
!
!
interface Vlan20
 description KSP
 ip address 192.168.110.1 255.255.255.224
 ip helper-address 192.168.101.215
!
router eigrp 1
 network 10.10.10.0
 network 192.168.96.0
 network 192.168.110.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.10.10.1
ip route 192.168.96.0 255.255.255.0 10.10.10.1
ip route 192.168.100.0 255.255.255.0 10.10.10.1
ip route 192.168.101.0 255.255.255.0 10.10.10.1
ip route 192.168.102.0 255.255.255.0 10.10.10.1
ip route 192.168.103.0 255.255.255.0 10.10.10.1
ip route 192.168.114.0 255.255.255.0 10.10.10.1
ip route 192.168.104.0 255.255.255.0 10.10.10.5
ip route 192.168.105.0 255.255.255.0 10.10.10.6
ip route 192.168.106.0 255.255.255.0 10.10.10.7
ip route 192.168.107.0 255.255.255.0 10.10.10.9
ip route 192.168.108.0 255.255.255.0 10.10.10.1
ip route 192.168.109.0 255.255.255.0 10.10.10.3
ip route 192.168.111.0 255.255.255.0 10.10.10.2
ip route 192.168.112.0 255.255.255.0 10.10.10.10
ip route 192.168.113.0 255.255.255.0 10.10.10.8
no ip http server
!
!
control-plane
!
!
line con 0
line vty 0 4
 password *****
 login
line vty 5 15
 password *****
 login
!
end

Open in new window

VLAN-Problem.jpg
0
Comment
Question by:CityofKerrville
  • 11
  • 7
18 Comments
 
LVL 57

Expert Comment

by:giltjr
ID: 22924176
What do you get if you issue show vtp status?
0
 

Author Comment

by:CityofKerrville
ID: 22924544
"What do you get if you issue show vtp status?"

KSP#sh vtp status
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 :
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x57 0xCD 0x40 0x65 0x63 0x59 0x47 0xBD
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 192.168.110.1 on interface Vl1 (lowest numbered VLAN interface found)
KSP#

Open in new window

0
 

Author Comment

by:CityofKerrville
ID: 22924562
Above was SITE D.

Below is SITE C


COURT#sh vtp stat
VTP Version                     : 2
Configuration Revision          : 4
Maximum VLANs supported locally : 1005
Number of existing VLANs        : 9
VTP Operating Mode              : Server
VTP Domain Name                 :
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0x50 0x19 0x94 0x2C 0x0A 0x69 0x61 0x74
Configuration last modified by 0.0.0.0 at 3-1-93 00:12:32
Local updater ID is 192.168.109.1 on interface Vl20 (lowest numbered VLAN interface found)
COURT#

Open in new window

0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 57

Expert Comment

by:giltjr
ID: 22924881
try issuing:

 set vlan 20 state active

Now, I have to think about it, but since the 3560's are L2/L3 devices if VLAN20 on both switches are not part of the same VLAN, then you should either have each of them have different VLAN's, or have them be part of two different VTP Domains.

I will need to look at your configs some more as something does not seem right about this.
0
 

Author Comment

by:CityofKerrville
ID: 22925029
try issuing:

 set vlan 20 state active

On which device should I do this.

SITE C is running on VLAN20 without fault

SITE D would not come up on VLAN20 or even on VLAN40 and is now running on VLAN1.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 22925236
Let me look at your configs some more.

Do you want Site C and Site D to be on the same VLAN and have traffic "switched" between the two?  Or do you want them to be seperate VLAN's and have all traffic routed between the two?
0
 

Author Comment

by:CityofKerrville
ID: 22925254
I would like them to be on the same VLAN and will want to add additional sites to the same VLAN in the future.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 22925510
What is the bandwidth on the Metro networks?

You do realize that  means that all broadcast traffic will be sent across the metro network to all sites?

I would suggest that you keep each site as its own VLAN (meaning each site will be its own IP subnet) so that you don't eat up the WAN traffic with broadcasts.

Unless there is some reason you want to do that.  It will be a 4-5 hours before I can respond again.
0
 

Author Comment

by:CityofKerrville
ID: 22925545
Each site is on its own ip subnet.  The purpose of the VLAN is to limit access to sensitive area's such as Police Department and Water Treatment Plants.
0
 
LVL 57

Expert Comment

by:giltjr
ID: 22927664
I agree that each site needs to be on its own subnet.  However, they also need to be on different VLAN's.

Even with each site having their own VLAN's I would also suggest that each site be part of its own VTP domain.  This way VLAN information is not exchanged between the switches at different sites.

The problem you are most likely having is that the 3560 are L2 and L3 devices and so they are exchanging L2 VTP information.  Where as it looks like the other sites have L3 only devices, so no VTP information is exchanged.
0
 

Author Comment

by:CityofKerrville
ID: 22930713
Ok, lets say for the sake of argument that we want to put everything on the same VTP domain?  Am I correct in assuming that we need to have only one VTP server?  How would we go about implemeting this.  Does our router at SITE A house the VLAN database?

Our ultimate goal is to secure the sensitve areas without locking us (meaning IT) out of anything.  here is a diagram that best illustrates our end goal.

Kerrville-Metro-Ethernet-VLANS.jpg
0
 

Author Comment

by:CityofKerrville
ID: 22930739
Right now were are only dealing with SITE A, B, C, and D.  My assumption is that once we have the configuration right.  We should just be able to plug in each site with out issue.
0
 

Author Comment

by:CityofKerrville
ID: 22930902
Here is the show vtp status for our main router at SITE A.  I believe this on should be our VTP server.

CHR1#en
CHR1#sh vtp stat
VTP Version                     : 2
Configuration Revision          : 0
Maximum VLANs supported locally : 52
Number of existing VLANs        : 5
VTP Operating Mode              : Server
VTP Domain Name                 :
VTP Pruning Mode                : Disabled
VTP V2 Mode                     : Disabled
VTP Traps Generation            : Disabled
MD5 digest                      : 0xBF 0x86 0x94 0x45 0xFC 0xDF 0xB5 0x70
Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00
Local updater ID is 192.168.96.1 on interface Vl10 (lowest numbered VLAN interface found)
CHR1#

Open in new window

0
 
LVL 57

Expert Comment

by:giltjr
ID: 22931464
If you wanted to have a single VTP domain, then yes, you would want to have only one VTP server.

However, you stated you wanted to route between sites across the metro network.  So VLAN20 at "SITE1" will NOT be the same VLAN20 at "SITE2".  This could be very confusing if you were to do this.  The connections to the metro network are not trunks, but are just "normal" single VLAN connections, except possibility in the sites where you have L3 switches.

From your diagram it looks like you wannt have VLAN20 at 10 or so sites.  However this will not be a single VLAN20, but 10 independent VLAN20's.  That would be very confusing, since normally VLAN's are a single broadcast domain and traffic within a VLAN is switched, not routed.

You should also have problems with your management VLAN10.  Since all of the switches/routers management IP address are within the same subnet, they will assume they can communicate directly with each other, no routing involved.  However, since the connections to the metro network are access mode ports, there is no switching only routing.  So you will have problems getting to the management IP addresses.
0
 

Author Comment

by:CityofKerrville
ID: 22932554
More confusion is certainly not needed.  Assuming we put each site on separate VLANS can you offer up a suggested solution that meets some the following criteria.

All sites able to talk to data center
All sites accessible to IT
Secure segregation of specific site (still keeping management access)

That would make 13 VLANS,
1 - Management (IT Workstations)
1 - Servers or Data Center (or should these be kept on the native?)
3 - Secure (only accessible to the user assigned to that VLAN, Servers, and Management)
8 - Regular users (workstations and printers)

I have a tenancy to over complicate things, so if there is an easier way to do what we want done please help me out.

Speaking of over complicating, I have attached new diagram.

Kerrville-Metro-Ethernet-VLANS-2.png
0
 
LVL 57

Expert Comment

by:giltjr
ID: 22949986
I have not forgotten about this.  I'm just thinking things through to make sure that I'm not overlooking things.  One suggestion is that you do NOT put your servers on the native VLAN.  Nothing should be put on native VLAN if at all possible.

Part of what makes this a bit complicated is that when typically when you have a WAN each site is considered a totally separate site and you can use the same VLAN number at each site.  VLAN information is not transmitted at all when using only routers.  

However, when using Metro Ethernet its not really a WAN, it is more like a LAN.  So instead of considering each site as a standalone network, you have to treat it as if it were one building and segmenting the whole building into smaller LAN's and that the "back bone" connection (the Metro Ethernet) is 'slow' and has high latency so you don't want to do L2 functions across the "back bone".

If you look at Cisco's 3 layer network architecture you will notice that they have access layer, distribution layer, and the core layer.  At the core they strongly suggest that you route (L3) between cores and to/from the distribution layer.  At the distribution layer you switch between to any distribution layers that are directly connected to each other and switch between the access layer.

In your setup you really don't have a core layer.  You are interconnecting the distribution layer (the switches/routers at each site) with each other.  So you want to route there to reduce traffic that is crossing over the Metro network.  You also are mixing switches and routers at the distribution layer which gives you a mixture of capabilities (L2/L3 switching vs. L3 routing only) which means you need to account for VTP on the devices that support it.
0
 

Author Comment

by:CityofKerrville
ID: 22954019
What would be the outcome if I turned VTP off all together.  I understand that it I do shut down VTP, VLAN20 at SITE C will never know that SITE also has a VLAN20.  I'm ok with that as long as everyone can talk to the servers and to the internet.  For the sake of argument, let say I turn off VTP on all devises.  Will this fix the current problem?
0
 

Accepted Solution

by:
CityofKerrville earned 0 total points
ID: 22987603
Resolved on my own by doing the following

1.     Set VTP mode to Transparent at each site
2.     Assigned a Unique  VTP Domain Name to each site
3.     Assigned each site to their own VLAN and Subnet

Put SITE E in server today with no problems.  will be making the change to the exiting site after hours.

Thanks for the help.
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In the hope of saving someone else's sanity... About a year ago we bought a Cisco 1921 router with two ADSL/VDSL EHWIC cards to load balance local network traffic over the two broadband lines we have, but we couldn't get the routing to work consi…
This article will show how Aten was able to supply easy management and control for Artear's video walls and wide range display configurations of their newsroom.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question