Cisco 3750 3650 switch SSH issue

My network infrastructure is set up with several 3750-Xs as my switch stack. From there I have several satellite 3650 switches that connect back to the core via two fiber pairs set up as a port channel. To manage my switches I have a separate management VLAN.

Here's my issue: while performing on something unrelated I noticed I could no longer SSH to one of my switches from either my linux machine nor my Win 7 machine (neither of these machines have an IP in the management VLAN). This switch trunks back to the core switch stack (like several others) and also has two switches that trunk into it to get back to the switch stack (they're "farther out" so to speak). I can ssh into those just fine, then ssh "back" into the switch I can no longer SSH into from my desktop machines.

At first I couldn't even SSH into the problem switch from other switches "closer" to the core switch stack including the core switch stack itself, then (through no change I made, and I'm the only one who should be working on these switches) suddenly I could.

Troubleshooting this further:
-I can ping all of the management IPs from my desktops besides the problem switch
-I can ping the problem switch's management IP from all of my switches, even when I couldn't connect into it from some of them
-SSH debug shows nothing helpful
-All switches have the same version of SSH
-Checking the allowed VLANs, the managment VLAN is allowed on the trunk heading to the problem switch from the core stack
-I keep versioning history on all of my switches and this switch's config hasn't been changed for at least 4 weeks, even then, any of the changes made to the problem switch or the core switch within the last year have had nothing to do SSH communications and I know I've SSH'd into the switch recently with no issue.

This one is a head scratcher for me. Any help is appreciated.
travisryanAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

lruiz52Commented:
Can you post a sanitized config of the problem switch and of one of the switches that you can ssh to from your desktop machine?
0
eeRootCommented:
During the times that you cannot SSH into this switch, can you ping it from your workstation and/or the core switch?  Can you verify that all of your switches have the same subnet for the switch management VLAN and default gateway set?
0
Luke SmithSenior Manager, Production EngineeringCommented:
If you can't ping the problem switch from the work station, the problem doesn't necessarily mean it is related to that switch. It sounds like a potential routing or cable issue.

Have you tried a different management IP on the problem switch or looked at the routing within your management VLAN?
0
Cloud Class® Course: Microsoft Windows 7 Basic

This introductory course to Windows 7 environment will teach you about working with the Windows operating system. You will learn about basic functions including start menu; the desktop; managing files, folders, and libraries.

travisryanAuthor Commented:
After doing some more troubleshooting on this it gets stranger, but more specific:
-The switch can ping the other switches via their management VLAN IPs
-It cannot ping any address on another VLAN, eventhough the all of the VLANs gateways are on the switch stack
-Traceroute yields no extra information
-No results for sh ip redirect
-From my Linux machine ssh -v ip address just shows "connection timeout"
0
travisryanAuthor Commented:
@eeRoot, all switches are set with a ip default-gateway

@Luke Smith, what do you mean by "routing within the management VLAN"?
0
travisryanAuthor Commented:
After more troubleshooting it looks like the problem switch can't even ping its own default gateway. It can ping the gateway of the management VLAN, but not it's own gateway. This is more confusing because there's several devices on this switch that can communicate with several subnets just fine, on top of the fact that two other switches sit behind/farther away from the core switch and they can communicate just fine.
0
Luke SmithSenior Manager, Production EngineeringCommented:
Now it is starting to sound like the VLAN db might be corrupt for the problem switch. From a device in the network, can you do a "show ip route" of the IP of the problem switch and does it show routes?
0
travisryanAuthor Commented:
@Luke

The sh ip route for the problem switch and all switches besides the core stack is blank. It shows the default gateway, a blank table, and ICMP redirect cache is empty.

is there a way to flush the VLAN db?
0
travisryanAuthor Commented:
@lruiz52 and Luke, below are sanitized configs for the problem switch and one of my working switches. VLAN 20 is for computers/servers, VLAN 200 is the management VLAN.
Problem-Switch-Clean.txt
Good-Switch-Clean.txt
0
eeRootCommented:
You seem to have two different IP ranges in use on VLAN 200

Good switch =  ip address 10.1.100.6 255.255.255.0
Problem switch =  ip address 20.1.200.7 255.255.255.0

Is VLAN 200 using 10.1.100.x or 20.1.200.x?  I'd assume the only one that works is the one defined in the core switch config.
0
travisryanAuthor Commented:
@eeRoot, this was a santiation issue. 20.1.200 is my management VLAN for this exercise. That should be 20.1.200.6
0
travisryanAuthor Commented:
After troubleshooting this further I've changed the default gateway to 20.1.200.1, which is the gateway for my Management VLAN. Apparently this a proxy arp issue. I still need help fixing it as I don't want all of my traffic flowing over the management VLAN
0
travisryanAuthor Commented:
Changing the default gateway on all of my switches to the management VLAN gateways was the solution. Thanks for everyone's input.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
travisryanAuthor Commented:
This was the best solution
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Switches / Hubs

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.