Link to home
Start Free TrialLog in
Avatar of Shark Attack
Shark Attack

asked on

ssh error

Would anyone know what the below error might mean? I have upgraded all the routers (about month ago) to newer IOS. I can ssh in just fine. I have over 30 routers all over different sites and they all have the same message. Any advice?

*May 25 02:58:04.638: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 04:29:10.350: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 07:39:40.449: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:27:04.528: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:27:43.552: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:28:35.168: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:29:13.596: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:29:53.200: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:30:32.504: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:32:01.128: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:33:20.820: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 08:51:09.315: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 11:11:47.514: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 12:16:04.546: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 12:24:36.770: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
Avatar of matrix8086
matrix8086
Flag of Romania image
There are automated pieces of software, written by hackers, who are trying to access your routers by force, the error is generated by wrong certificate private key that the hackers are trying to guess or trick your routers.

This attempts were for ever, but you didn't see them with the old IOS

I always change the default 22 ssh port, with another one, for every device that is connected to the Internet (has an IP public address)

Best regards!
Avatar of Rafael
Rafael
Flag of United States of America image
Since you upgraded the IOS the crypto key may have changed. So a ssh connection towards this device is trying with the old known host key which might be causing the problem if it's cached somewhere. You can do delete the known host in the ssh client and try to do a clean ssh which should solve the issue.

[i]HTH
-Rafael [/i]
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
If this is a public facing router (accessible from the Internet for example) it could be outsiders trying to bruteforce their way into the router.

It could be an application (e.g monitoring) that tries to login and it doesn't work anymore because either rsa keys changed or because of the upgrade something isn't working anymore.
Avatar of Rafael
Rafael
Flag of United States of America image
I would say that's true as well. However, it started happening after the IOS upgrade and across multiple routers across the network. I would still look at the Crypto Key first then look at what Syslog is saying.
Avatar of Shark Attack
Shark Attack

ASKER

right, if this was on 1 or 2 routers its different story but when you have all 30 something routers doing the same thing it must be the crypto key. Should I just re-generate?
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
Yeah, logical thing would be to try the crypto-keys first.
Avatar of Shark Attack
Shark Attack

ASKER

Ok, do I have to somehow clear or delete the current or will it do that on it's own once I regenerate?
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
You can replace the key yes ... Something like this I think should do it:
crypto key generate rsa modulus 2048 label new_key
ip ssh rsa keypair-name new_key

Open in new window

Avatar of Shark Attack
Shark Attack

ASKER

thank you, I will keep everyone posted soon as I know
Avatar of Rafael
Rafael
Flag of United States of America image
As I mentioned in the initial posting response the crypto key more than likely is the culprit given it goes across multiple routers.  Especially if you did not define a domain-name during the upgrade.  However, you need to make sure you keep the same baseline across all your devices. This will avoid future problems.  Also, you will need to make sure you know what type of Key you're wanting.

Before you generate them you should execute the following:

router#sh ssh
router#sh domain-name
router#show crypto key mypubkey rsa

Here is a great "How To"  http://confterminal.com/configure-ssh-on-a-cisco-router/

HTH
-Rafael
Avatar of Shark Attack
Shark Attack

ASKER

seems like it didn't work. I think that it just added a key instead of replacing a key.
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
Check if your key is listed:

show crypto key mypubkey rsa

Open in new window


To set your key to use it should really be:

p ssh rsa keypair-name new_key

Open in new window


So, you're really saying which key to use.
Avatar of Shark Attack
Shark Attack

ASKER

i don't get the second command. I cant input that command in

The new key was listed. I do feel like it was just added though
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
The last command is actually:

ip ssh rsa keypair-name <name-of-key>

Open in new window



Where name-of-key is the name of the new key you created.

This command needs to be entered in config mode by the way ...
Avatar of Shark Attack
Shark Attack

ASKER

ok looks good now, I will let you know tomorrow again. thanks
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
Great, thanks!
Avatar of Rafael
Rafael
Flag of United States of America image
Don't forget to check it again after you are done.

router#show crypto key mypubkey rsa

HTH
-Rafael
Avatar of Shark Attack
Shark Attack

ASKER

still getting the errors. when I did the show crypto key mypubkey rsa, i do see the key created

Key pair was generated at: 16:33:37 UTC Jun 2 2015
Key name: for_ssh
Key type: RSA KEYS
 Storage Device: not specified
 Usage: General Purpose Key
 Key is not exportable.
 Key Data:
Avatar of Rafael
Rafael
Flag of United States of America image
Have you rebooted your device after creating the keys ? What does your logs say ?

[i]HTH
-Rafael[/i]
Avatar of Shark Attack
Shark Attack

ASKER

I did not reboot the device. 

*Jun  3 00:26:19.758: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  3 01:47:16.414: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  3 02:11:53.674: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  3 14:09:58.651: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
Suzlon_Amherst_Nova_Scotia_CA_Router#

Open in new window

Avatar of Rafael
Rafael
Flag of United States of America image
Are you able to reboot it after executing the command. I know that it's a router so you will have to schedule that outage.
Avatar of Shark Attack
Shark Attack

ASKER

yeah, I will reboot it tonight, will keep you posted thanks
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
Weird, hope the reboot will help because normally it shouldn't be necessary...
Avatar of Shark Attack
Shark Attack

ASKER

well, i don't see the error today but I think I will wait one more day and see what happens. thanks for all your help guys. i will keep everyone posted tomorrow.
Avatar of Rafael
Rafael
Flag of United States of America image
No problem. Keep us posted.

-Rafael
Avatar of Shark Attack
Shark Attack

ASKER

looks like one came in again late yesterday on top of some new errors related to ssh now

*Jun  4 20:55:50.923: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  4 20:56:31.167: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  4 20:57:37.491: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  5 05:32:27.121: %SSH-3-PACK_SND_FAIL: Packet send failed
Avatar of Rafael
Rafael
Flag of United States of America image
Was it on the same device that you rebooted ?
Avatar of Shark Attack
Shark Attack

ASKER

yes
Avatar of Rafael
Rafael
Flag of United States of America image
Are you able to do a debug?  Would like to to a debug  for ssh, attempt access, and post the debug output for us.

Also...Before you lost the config and the SSH access was working was it configured for aaa? I am curious if the issue is that SSH wants both a user name and a password but the default authentication on the vty ports only uses a password.
Are you able to post a clean sh run config from the device ? Just attach it as a text file.

HTH
-Rafael
Avatar of Shark Attack
Shark Attack

ASKER

the ssh is currently working, i just dont know why i'm getting those loggs. we do have ACS and this device is part of the devices in ACS, yea it;s also set up with aaa. here is the config. let me know if you still need a debug, ssh is working so i doubt i get any outputs, i aslo attached show crypto key thanks
show-cry-key.txt
router.txt
Avatar of Shark Attack
Shark Attack

ASKER

*Jun  6 16:59:09.104: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  6 22:56:08.862: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  7 17:21:47.170: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  7 23:49:14.092: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 01:38:01.612: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 09:41:11.594: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 10:17:54.386: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 11:26:35.377: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
It has to be something that is causing this, maybe you should implement access rules to allow only a few IP-addresses access to the router(s)... This way you might be able to pinpoint the troublemaker?

Something like this:
access-list 10 permit <Allowed1>
access-list 10 permit <Allowed2>
line vty 0 15
 access-class 10 in
 transport input ssh

Open in new window

Avatar of Shark Attack
Shark Attack

ASKER

Sorry about the late response, too much going on. I applied it. Will see what happens, I will let you know. thanks
Avatar of Shark Attack
Shark Attack

ASKER

well, logs are clear. I guess i just needed to put some restrictions on the vty lines.
ASKER CERTIFIED SOLUTION
Avatar of Zephyr ICT
Zephyr ICT
Flag of Belgium image
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Shark Attack
Shark Attack

ASKER

thanks!