I just upgraded a stack of 3850s to 3.7.4. Â After rebooting, the ten gig port (te1/1/4) in the first member of the stack is flashing amber. Â I look through the logs and found this error:
"Warning: adding member TenGigabitEthernet3/1/4 without policy will cause inconsistency with port TenGigabitEthernet1/1/4 in ether channel 1"
I checked, and from what I can tell both ports are part of the same channel group. Â Port te3/1/4 is up with no issue.
There are connecting back to Cisco 6880s.
version 15.2
no service pad
service timestamps debug datetime localtime
service timestamps log datetime localtime
service password-encryption
service compress-config
!
hostname xxx
!
boot-start-marker
boot system switch all flash:packages.conf
boot-end-marker
!
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
logging buffered 214748364
enable secret xxx
!
username admin privilege 15 password 7 xxx
software auto-upgrade enable
!
aaa new-model
!
!
aaa authentication login default local
!
!
!
!
!
!
aaa session-id common
clock timezone EST -5 0
clock summer-time EDT recurring
switch 1 provision ws-c3850-12x48u
switch 2 provision ws-c3850-48p
switch 3 provision ws-c3850-48p
!
stack-power stack 1
!
!
!
!
!
!
coap http enable
!
!
!
!
!
!
!
no ip domain-lookup
ip domain-name lourdes.edu
!
!
qos queue-softmax-multiplier 100
vtp mode transparent
udld enable
Port-Channel appears to be configured ok. Did the upgrade succeed throughout the stack?
What does
show etherchannel 1 detail
Report?
Pete Long
May not be relevant - but I had a colleague had to DOWNGRADE some 3850's last week due to strange stacking errors
P
Gubbio
ASKER
show etherchannel 1 detail
Group state = L2
Ports: 2 Â Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: Â LACP
Minimum Links: 0
        Ports in the group:
        -------------------
Port: Te1/1/4
------------
Port state   = Up Mstr Assoc Not-in-Bndl
Channel group = 1      Mode = Active      Gcchange = -
Port-channel  = Po1     GC  =  -       Pseudo port-channel = Po1
Port index   = 0      Load = 0x00       Protocol =  LACP
Flags:  S - Device is sending Slow LACPDUs  F - Device is sending fast LACPDUs.
    A - Device is in active mode.     P - Device is in passive mode.
Local information:
              LACP port   Admin   Oper   Port     Port
Port    Flags  State   Priority    Key    Key   Number    State
Te1/1/4  SA    bndl    32768     0x1    0x1   0x139    0xD
Partner's information:
         LACP port             Admin  Oper  Port   Port
Port    Flags  Priority  Dev ID      Age   key   Key   Number  State
Te1/1/4  SA    32768   0200.0000.0001  22s   0x0   0x6B  0x1503  0x75
Age of the port in the current state: 0d:00h:00m:19s
Port: Te3/1/4
------------
Port state   = Up Mstr Assoc In-Bndl
Channel group = 1      Mode = Active      Gcchange = -
Port-channel  = Po1     GC  =  -       Pseudo port-channel = Po1
Port index   = 0      Load = 0x00       Protocol =  LACP
Flags:  S - Device is sending Slow LACPDUs  F - Device is sending fast LACPDUs.
    A - Device is in active mode.     P - Device is in passive mode.
Local information:
              LACP port   Admin   Oper   Port     Port
Port    Flags  State   Priority    Key    Key   Number    State
Te3/1/4  SA    bndl    32768     0x1    0x1   0x339    0x3D
Partner's information:
         LACP port             Admin  Oper  Port   Port
Port    Flags  Priority  Dev ID      Age   key   Key   Number  State
Te3/1/4  SA    32768   0200.0000.0001  15s   0x0   0x6B  0x2503  0x3D
Age of the port in the current state: 2d:19h:59m:35s
        Port-channels in the group:
        ---------------------------
Port-channel: Po1 Â Â (Primary Aggregator)
------------
Age of the Port-channel  = 2d:19h:59m:56s
Logical slot/port  = 12/1      Number of ports = 2
HotStandBy port = null
Port state      = Port-channel Ag-Inuse
Protocol       =  LACP
Port security    = Disabled
Ports in the Port-channel:
Index  Load  Port   EC state     No of bits
------+------+------+------------------+-----------
 0   00   Te1/1/4  Active       0
 0   00   Te3/1/4  Active       0
Time since last port bundled:   0d:00h:00m:19s   Te1/1/4
Time since last port Un-bundled: 0d:00h:00m:22s   Te1/1/4
Ok, so the bundel forms and destroys the bundle in msec.
Can you enable 'debug port-channel error' for a few seconds to see what is wrong?
Oh btw, is the channel now working on the 3/1/4?
Predrag Jovic
Did you upgrade just 1 or all 3850s (are IOS version matching, snce you have 2 different switch models in the stack)?
Issue:
show version
Gubbio
ASKER
Port 3/1/4 is working.
I updated the code to 03.07.04, which is Cisco's recommended for have 3850 MGIG and standard 3850s in a stack together.
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.04E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Thu 19-May-16 11:48 by prod_rel_team
Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Â Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. Â The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. Â You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
stfrsw1 uptime is 3 days, 3 minutes
Uptime for this control processor is 3 days, 7 minutes
System returned to ROM by reload at 15:17:24 EDT Fri Mar 31 2017
System restarted at 15:37:41 EDT Fri Mar 31 2017
System image file is "flash:packages.conf"
Last reload reason: snmp shutdown request
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
cisco WS-C3850-12X48U (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FCW1921C05T
1 Virtual Ethernet interface
144 Gigabit Ethernet interfaces
28 Ten Gigabit Ethernet interfaces
2 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
257008K bytes of Crash Files at crashinfo:.
250456K bytes of Crash Files at crashinfo-2:.
250456K bytes of Crash Files at crashinfo-3:.
1550272K bytes of Flash at flash:.
1609272K bytes of Flash at flash-2:.
1609272K bytes of Flash at flash-3:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of Dummy USB Flash at usbflash0-2:.
0K bytes of Dummy USB Flash at usbflash0-3:.
0K bytes of  at webui:.
Switch Ports Model        SW Version     SW Image        Mode
------ ----- ----- Â Â Â Â Â Â Â ---------- Â Â Â Â ---------- Â Â Â Â Â Â ----
* Â Â 1 62 Â Â WS-C3850-12X48U Â Â 03.07.04E Â Â Â Â cat3k_caa-universalk9 INSTALL
   2 56   WS-C3850-48P    03.07.04E     cat3k_caa-universalk9 INSTALL
   3 56   WS-C3850-48P    03.07.04E     cat3k_caa-universalk9 INSTALL
Did you try a debugging port-channel? I am curious why the adjacencie Isnt being formed.
I also would like to know what is going on with the 6880s, Maybe the problem is not locally but in forming the LACP link. Can you show the particulair config for the remote ports?
Cdp neighbor shows a 6880 for the 1/1/4 is assume?
Gubbio
ASKER
Links on 6880 look fine. Â
On the 3850s, te1/1/4 says "waiting to be aggregated".
How do debug port-channel?
Patrick Bogers
Ah you read over my previous question, can happen.
Debugging port channel like. Â debug port-channel error
If this doesnt show any clubs i would take-out 1/1/4 out off the etherchannel and rebuild the channel.
Apr  4 15:55:21: FEC: delete port (Te1/1/4) from agport (Po1)
Apr  4 15:55:21: FEC: lacp_switch_remove_port_from_associated_list_internal: Te1/1/4 deleted from the associated list for Po1
Apr  4 15:55:24: FEC: add port (Te1/1/4) to agport (Po1)
Apr  4 15:55:47: FEC: delete port (Te1/1/4) from agport (Po1)
Apr  4 15:55:47: FEC: lacp_switch_remove_port_from_associated_list_internal: Te1/1/4 deleted from the associated list for Po1
Apr  4 15:55:51: FEC: add port (Te1/1/4) to agport (Po1)
Apr  4 15:56:13: FEC: delete port (Te1/1/4) from agport (Po1)
Apr  4 15:56:13: FEC: lacp_switch_remove_port_from_associated_list_internal: Te1/1/4 deleted from the associated list for Po1
Apr  4 15:56:17: FEC: add port (Te1/1/4) to agport (Po1)
Apr  4 15:56:41: FEC: lacp_switch_delete_port_from_agport_internal: removing Te1/1/4 from Po1
Apr  4 15:56:41: FEC: delete port (Te1/1/4) from agport (Po1)
Apr  4 15:56:41: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 1 [Te1/1/4]
Apr  4 15:56:41: FEC: lacp_switch_remove_port_from_associated_list_internal: Te1/1/4 deleted from the associated list for Po1
Apr  4 15:56:41: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:41: FEC: pagp_switch_reset_load_index: reading load-index for port Po1
Apr  4 15:56:41: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:41: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  4 15:56:41: FEC: lacp_switch_is_port_in_associate_list: port Te3/1/4 is present in the associate list
Apr  4 15:56:41: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Te3/1/4 in aggregator Po1
Apr  4 15:56:41: FEC: lacp_switch_get_first_agg_id_from_admin_oper_keys: found aggregator Po1 for admin_key [0x1] and oper_key [0x1]
Apr  4 15:56:41: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:41: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  4 15:56:41: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:41: FEC: lacp_switch_add_port_to_associated_list_internal: Te1/1/4 added to list for Po1
Apr  4 15:56:41: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/4 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  4 15:56:41: FEC: lacp_switch_agport_fast_switchover_enabled: Te1/1/4 lacp fast switchover disabled
Apr  4 15:56:43: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:43: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/4 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  4 15:56:43: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:43: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  4 15:56:43: FEC: lacp_switch_is_port_in_associate_list: port Te3/1/4 is present in the associate list
Apr  4 15:56:43: FEC: lacp_switch_get_next_associated_port_from_agg_id: found port Te1/1/4 next to Te3/1/4 and associated to Po1
Apr  4 15:56:43: FEC: lacp_switch_is_port_in_associate_list: port Te1/1/4 is present in the associate list
Apr  4 15:56:43: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Te1/1/4 in aggregator Po1
Apr  4 15:56:43: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:43: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/4 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  4 15:56:43: FEC: lacp_switch_agport_fast_switchover_enabled: Te1/1/4 lacp fast switchover disabled
Apr  4 15:56:45: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:45: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/4 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  4 15:56:45: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:45: FEC: add port (Te1/1/4) to agport (Po1)
Apr  4 15:56:45: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 2 [Te1/1/4]
Apr  4 15:56:45: FEC: lacp_switch_add_port_to_agport_internal: Te1/1/4 added to aggregator Po1 list
Apr  4 15:56:45: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  4 15:56:45: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  4 15:56:45: FEC: lacp_switch_is_port_in_associate_list: port Te3/1/4 is present in the associate list
Apr  4 15:56:45: FEC: lacp_switch_get_next_associated_port_from_agg_id: found port Te1/1/4 next to Te3/1/4 and associated to Po1
Apr  4 15:56:45: FEC: lacp_switch_is_port_in_associate_list: port Te1/1/4 is present in the associate list
Apr  4 15:56:45: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Te1/1/4 in aggregator Po1
Apr  4 15:57:09: FEC: lacp_switch_delete_port_from_agport_internal: removing Te1/1/4 from Po1
Gubbio
ASKER
If I have to rebuild the port-channel, can I do so without causing an outage?
Patrick Bogers
Well Maybe rebuild was a incorrect word. I ment to delete 1/1/4 as a port channel member, reset the interface to a default interface and reconfigure it to become a member off the port channel 1 again.
Basically the same now happens in the debugging output. The channel stays but only with one port part of the bundle.
It might also be an idea to use another port than the 1/1/4 (if you have one vacant)
Alright. Â I have removed te1/1/4 from the port-channel, reset it to default config, configured te1/1/3, added it to the port-channel, and I am still get errors.
Apr  5 14:58:26: %LINK-5-CHANGED: Interface TenGigabitEthernet1/1/3, changed state to administratively down
Apr  5 14:58:48: FEC: pm_update_agport_member_cbs: Te1/1/3 not an agport
Apr  5 14:58:48: Warning: adding member TenGigabitEthernet1/1/3 without policy will cause inconsistency with port TenGigabitEthernet3/1/4 in ether channel 1.
Apr  5 14:58:48: FEC: fec_user_enable_internal: port = Te1/1/3 group = 1 mode = 6
Apr  5 14:58:48: FEC: add port Te1/1/3 to group 1
Apr  5 14:58:48: FEC: pagp_switch_enable: Te1/1/3
Apr  5 14:58:48: FEC: pagp_switch_enable: enabling LACP on Te1/1/3
Apr  5 14:58:48: Warning: adding member TenGigabitEthernet1/1/3 without policy will cause inconsistency with port TenGigabitEthernet3/1/4 in ether channel 1. (stxxxx)
Apr  5 14:59:37: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.x.x.x)
Apr  5 14:59:47: FEC: lacp_switch_display_oneline: found 1 aggregators
Apr  5 14:59:47: FEC: lacp_switch_display_oneline: found 2 ports
Apr  5 15:00:19: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to down
Apr  5 15:00:26: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.x.x.x)
Apr  5 15:00:42: FEC: lacp_switch_display_oneline: found 1 aggregators
Apr  5 15:00:42: FEC: lacp_switch_display_oneline: found 2 ports
Apr  5 15:05:45: %PLATFORM_PM-6-MODULE_REMOVED: SFP module with interface name Te1/1/4 removed
Apr  5 15:05:52: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3
Apr  5 15:05:52: FEC: pagp_switch_port_attrib_change: change for Te1/1/3
Apr  5 15:05:52: FEC: pagp_switch_port_attrib_change: port Te1/1/3 is down
Apr  5 15:06:01: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to up
Apr  5 15:06:02: FEC: pagp_switch_port_up: Te1/1/3
Apr  5 15:06:02: FEC: pagp_switch_invoke_port_up: Te1/1/3
Apr  5 15:06:02: FEC: pagp_switch_agc_compatable: comparing GC values of Te1/1/3 Te3/1/4 flag = 1 1
Apr  5 15:06:02: FEC: pagp_switch_port_attrib_diff: compare LACP modes for Te1/1/3
Apr  5 15:06:02: FEC: pagp_switch_port_attrib_diff: Te1/1/3 Te3/1/4 same
Apr  5 15:06:02: FEC: pagp_switch_agc_compatable: GC values are compatable
Apr  5 15:06:02: FEC: lacp_switch_calculate_oper_key: oper_key for port Te1/1/3 is 0x1
Apr  5 15:06:03: FEC: lacp_switch_get_first_agg_id_from_admin_oper_keys: found aggregator Po1 for admin_key [0x1] and oper_key [0x1]
Apr  5 15:06:03: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:03: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  5 15:06:03: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:03: FEC: lacp_switch_add_port_to_associated_list_internal: Te1/1/3 added to list for Po1
Apr  5 15:06:03: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/3 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  5 15:06:03: FEC: lacp_switch_agport_fast_switchover_enabled: Te1/1/3 lacp fast switchover disabled
Apr  5 15:06:04: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:04: FEC: lacp_switch_remove_port_from_associated_list_internal: Te1/1/3 deleted from the associated list for Po1
Apr  5 15:06:04: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:04: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  5 15:06:04: FEC: lacp_switch_is_port_in_associate_list: port Te3/1/4 is present in the associate list
Apr  5 15:06:04: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Te3/1/4 in aggregator Po1
Apr  5 15:06:04: FEC: lacp_switch_get_first_agg_id_from_admin_oper_keys: found aggregator Po1 for admin_key [0x1] and oper_key [0x1]
Apr  5 15:06:04: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:04: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  5 15:06:04: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:04: FEC: lacp_switch_add_port_to_associated_list_internal: Te1/1/3 added to list for Po1
Apr  5 15:06:04: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/3 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  5 15:06:04: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:04: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/3 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  5 15:06:04: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:04: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  5 15:06:04: FEC: lacp_switch_is_port_in_associate_list: port Te3/1/4 is present in the associate list
Apr  5 15:06:04: FEC: lacp_switch_get_next_associated_port_from_agg_id: found port Te1/1/3 next to Te3/1/4 and associated to Po1
Apr  5 15:06:04: FEC: lacp_switch_is_port_in_associate_list: port Te1/1/3 is present in the associate list
Apr  5 15:06:04: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Te1/1/3 in aggregator Po1
Apr  5 15:06:04: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:04: FEC: lacp_switch_check_hw_sw_constraints_internal: port Te1/1/3 can be bundled in the aggregator Po1, new afb->nports [1]
Apr  5 15:06:04: FEC: lacp_switch_agport_fast_switchover_enabled: Te1/1/3 lacp fast switchover disabled
Apr  5 15:06:05: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:05: FEC: lacp_switch_remove_port_from_associated_list_internal: Te1/1/3 deleted from the associated list for Po1
Apr  5 15:06:05: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Apr  5 15:06:05: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Te3/1/4 associated to Po1
Apr  5 15:06:05: FEC: lacp_switch_is_port_in_associate_list: port Te3/1/4 is present in the associate list
Apr  5 15:06:05: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Te3/1/4 in aggregator Po1
Apr  5 15:06:05: FEC: lacp_switch_get_first_agg_id_from_admin_oper_keys: found aggregator Po1 for admin_key [0x1] and oper_key [0x1]
Apr  5 15:06:05: FEC: lacp_switch_is_aggregator_valid: aggregator Po1 is still valid
Port-Channel appears to be configured ok. Did the upgrade succeed throughout the stack?
What does
show etherchannel 1 detail
Report?