ssh error

Would anyone know what the below error might mean? I have upgraded all the routers (about month ago) to newer IOS. I can ssh in just fine. I have over 30 routers all over different sites and they all have the same message. Any advice?

*May 25 02:58:04.638: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 04:29:10.350: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 07:39:40.449: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:27:04.528: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:27:43.552: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:28:35.168: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:29:13.596: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:29:53.200: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:30:32.504: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:32:01.128: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 25 12:33:20.820: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 08:51:09.315: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 11:11:47.514: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 12:16:04.546: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*May 26 12:24:36.770: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
LVL 3
Shark AttackNetwork adminAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

matrix8086Commented:
There are automated pieces of software, written by hackers, who are trying to access your routers by force, the error is generated by wrong certificate private key that the hackers are trying to guess or trick your routers.

This attempts were for ever, but you didn't see them with the old IOS

I always change the default 22 ssh port, with another one, for every device that is connected to the Internet (has an IP public address)

Best regards!
0
RafaelCommented:
Since you upgraded the IOS the crypto key may have changed. So a ssh connection towards this device is trying with the old known host key which might be causing the problem if it's cached somewhere. You can do delete the known host in the ssh client and try to do a clean ssh which should solve the issue.

[i]HTH
-Rafael [/i]
0
Zephyr ICTCloud ArchitectCommented:
If this is a public facing router (accessible from the Internet for example) it could be outsiders trying to bruteforce their way into the router.

It could be an application (e.g monitoring) that tries to login and it doesn't work anymore because either rsa keys changed or because of the upgrade something isn't working anymore.
0
Get Cisco Certified in IT Security

There’s a high demand for IT security experts and network administrators who can safeguard the data that individuals, corporations, and governments rely on every day. Pursue your B.S. in Network Operations and Security and gain the credentials you need for this high-growth field.

RafaelCommented:
I would say that's true as well. However, it started happening after the IOS upgrade and across multiple routers across the network. I would still look at the Crypto Key first then look at what Syslog is saying.
0
Shark AttackNetwork adminAuthor Commented:
right, if this was on 1 or 2 routers its different story but when you have all 30 something routers doing the same thing it must be the crypto key. Should I just re-generate?
0
Zephyr ICTCloud ArchitectCommented:
Yeah, logical thing would be to try the crypto-keys first.
0
Shark AttackNetwork adminAuthor Commented:
Ok, do I have to somehow clear or delete the current or will it do that on it's own once I regenerate?
0
Zephyr ICTCloud ArchitectCommented:
You can replace the key yes ... Something like this I think should do it:
crypto key generate rsa modulus 2048 label new_key
ip ssh rsa keypair-name new_key

Open in new window

0
Shark AttackNetwork adminAuthor Commented:
thank you, I will keep everyone posted soon as I know
0
RafaelCommented:
As I mentioned in the initial posting response the crypto key more than likely is the culprit given it goes across multiple routers.  Especially if you did not define a domain-name during the upgrade.  However, you need to make sure you keep the same baseline across all your devices. This will avoid future problems.  Also, you will need to make sure you know what type of Key you're wanting.

Before you generate them you should execute the following:

router#sh ssh
router#sh domain-name
router#show crypto key mypubkey rsa

Here is a great "How To"  http://confterminal.com/configure-ssh-on-a-cisco-router/

HTH
-Rafael
0
Shark AttackNetwork adminAuthor Commented:
seems like it didn't work. I think that it just added a key instead of replacing a key.
0
Zephyr ICTCloud ArchitectCommented:
Check if your key is listed:

show crypto key mypubkey rsa

Open in new window


To set your key to use it should really be:

p ssh rsa keypair-name new_key

Open in new window


So, you're really saying which key to use.
0
Shark AttackNetwork adminAuthor Commented:
i don't get the second command. I cant input that command in

The new key was listed. I do feel like it was just added though
0
Zephyr ICTCloud ArchitectCommented:
The last command is actually:

ip ssh rsa keypair-name <name-of-key>

Open in new window



Where name-of-key is the name of the new key you created.

This command needs to be entered in config mode by the way ...
0
Shark AttackNetwork adminAuthor Commented:
ok looks good now, I will let you know tomorrow again. thanks
0
Zephyr ICTCloud ArchitectCommented:
Great, thanks!
0
RafaelCommented:
Don't forget to check it again after you are done.

router#show crypto key mypubkey rsa

HTH
-Rafael
0
Shark AttackNetwork adminAuthor Commented:
still getting the errors. when I did the show crypto key mypubkey rsa, i do see the key created

Key pair was generated at: 16:33:37 UTC Jun 2 2015
Key name: for_ssh
Key type: RSA KEYS
 Storage Device: not specified
 Usage: General Purpose Key
 Key is not exportable.
 Key Data:
0
RafaelCommented:
Have you rebooted your device after creating the keys ? What does your logs say ?

[i]HTH
-Rafael[/i]
0
Shark AttackNetwork adminAuthor Commented:
I did not reboot the device.

*Jun  3 00:26:19.758: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  3 01:47:16.414: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  3 02:11:53.674: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  3 14:09:58.651: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
Suzlon_Amherst_Nova_Scotia_CA_Router#               

Open in new window

0
RafaelCommented:
Are you able to reboot it after executing the command. I know that it's a router so you will have to schedule that outage.
0
Shark AttackNetwork adminAuthor Commented:
yeah, I will reboot it tonight, will keep you posted thanks
0
Zephyr ICTCloud ArchitectCommented:
Weird, hope the reboot will help because normally it shouldn't be necessary...
0
Shark AttackNetwork adminAuthor Commented:
well, i don't see the error today but I think I will wait one more day and see what happens. thanks for all your help guys. i will keep everyone posted tomorrow.
0
RafaelCommented:
No problem. Keep us posted.

-Rafael
0
Shark AttackNetwork adminAuthor Commented:
looks like one came in again late yesterday on top of some new errors related to ssh now

*Jun  4 20:55:50.923: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  4 20:56:31.167: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  4 20:57:37.491: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  5 05:32:27.121: %SSH-3-PACK_SND_FAIL: Packet send failed
0
RafaelCommented:
Was it on the same device that you rebooted ?
0
Shark AttackNetwork adminAuthor Commented:
yes
0
RafaelCommented:
Are you able to do a debug?  Would like to to a debug  for ssh, attempt access, and post the debug output for us.

Also...Before you lost the config and the SSH access was working was it configured for aaa? I am curious if the issue is that SSH wants both a user name and a password but the default authentication on the vty ports only uses a password.
Are you able to post a clean sh run config from the device ? Just attach it as a text file.

HTH
-Rafael
0
Shark AttackNetwork adminAuthor Commented:
the ssh is currently working, i just dont know why i'm getting those loggs. we do have ACS and this device is part of the devices in ACS, yea it;s also set up with aaa. here is the config. let me know if you still need a debug, ssh is working so i doubt i get any outputs, i aslo attached show crypto key thanks
show-cry-key.txt
router.txt
0
Shark AttackNetwork adminAuthor Commented:
*Jun  6 16:59:09.104: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  6 22:56:08.862: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  7 17:21:47.170: %SSH-3-PACK_SND_FAIL: Packet send failed
*Jun  7 23:49:14.092: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 01:38:01.612: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 09:41:11.594: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 10:17:54.386: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
*Jun  8 11:26:35.377: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
0
Zephyr ICTCloud ArchitectCommented:
It has to be something that is causing this, maybe you should implement access rules to allow only a few IP-addresses access to the router(s)... This way you might be able to pinpoint the troublemaker?

Something like this:
access-list 10 permit <Allowed1>
access-list 10 permit <Allowed2>
line vty 0 15
 access-class 10 in
 transport input ssh

Open in new window

0
Shark AttackNetwork adminAuthor Commented:
Sorry about the late response, too much going on. I applied it. Will see what happens, I will let you know. thanks
0
Shark AttackNetwork adminAuthor Commented:
well, logs are clear. I guess i just needed to put some restrictions on the vty lines.
0
Zephyr ICTCloud ArchitectCommented:
Maybe someday something will pop out that will clear things up :)
Nice to hear the logs are clear for now ...
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Shark AttackNetwork adminAuthor Commented:
thanks!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Routers

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.