I have a bunch of Procurve Switches; it's a complex multiple-VLAN network. Had an issue today with two switches that I have never seen before, and has me perplexed.
Our VLAN 30 runs from our core switch, a 5406zl, to the "uplink" switch at the location that is VLAN 30. The "uplink" switch is a 2824. From it, VLAN 30 is extended to two more switches, a 2848 ("clubhouse") and a 2626 ("maintenance"). VLAN 30 is tagged on each of the link ports; D1 on the 5406zl, 22-24 on the "uplink" to receive from the 5406zl (24) and send to the "clubhouse" (22) and "maintenance" (23). And to complete the design, VLAN 30 is then tagged on port 48 of "clubhouse" and port 26 of "maintenance" to receive the VLAN on those switches. The rest of the ports on "uplink", "clubhouse", and "maintenance" are untagged in VLAN 30, taking client connections. There are other VLANs at play here, but are not relevant to the discussion.
So here's where it gets weird. Going into today, the configurations were tested, fully working, and committed to memory (wr mem). These switches are all in production, everything was working fine. Late morning, with no explainable cause, many devices were inaccessible on VLAN 30. Ultimately, it was found that "clubhouse" port 48 was no longer tagged in VLAN 30. It was still tagged on port 26 of "maintenance" and ports 22-24 of "uplink", and D1 of the 5406zl.
In testing, we moved the "clubhouse" switch physically closer to the "uplink" switch, and also used copper instead of fiber to cross-connect (easier when they're two feet apart). I repaired the configuration by tagging 48 into VLAN 30 again, and wrote the configuration. When the switch was powered off and moved back to its production location, fiber cables and GBICs reconnected, VLAN 30 was no longer tagged on port 48!!!
I have tagged it again there and wrote the configuration, and things are working, but I fully expect a power outage to undo the change.
The "clubhouse" 2848 switch is running version I.08.98, if this is perhaps part of the problem.
Thanks for any assistance.