MKS Error on two hosts in a four host cluster - vSphere 5.5

Could not create console connection to certain VMs either through vSphere client or web client with Error message: Unable to connect to MKS: A general system error occurred: Internal error
After investigating some more it was only VMs on two of the hosts in a four host cluster.  Enabled SSH on one of the affected hosts to restart mgmt services but got error message "Network error: Connection refused". Same on the other affected host but could connect with SSH to the other hosts with no issue.
Tried from vCenter server but same result. Rebooted vCenter server but same result.

Logged on to console session on both dodgy hosts and did management services restart - still cannot ssh to them and still cannot do a console session to VMs, however error message changed to Error Message: Unable to connect to server IPAddress:902
Logged on to shell and did a vdf -h but no issues with full root ramdisk or anything obvious.

Also cannot vmotion anything off the affected hosts and would prefer not to have to shut doen the VMs (which appear to be working fine) so any ideas or pointers please?   Will log with VMWare support in the morning but like to fix myself if possible.
LVL 1
nchtIT Operations ManagerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If restarting management agents is not helping.

Shutdown VMs, and restart the Host.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Paul SolovyovskySenior IT AdvisorCommented:
also try logging directly to the hosts, vs vcenter via vi client.
nchtIT Operations ManagerAuthor Commented:
Some of the VMs hosts part of a large SharePoint farm so do not really want to shut down VMs if possible.  Also connects Ok to host direct with client but still same issues on VMs.
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If you have a fault, you will need to schedule emergency downtime, is your Sharepoint Farm not designed with failure in the design ?
nchtIT Operations ManagerAuthor Commented:
Andrew. Thanks for taking the time to reply.  SharePoint has failover but was just trying to avoid but looking like we will have to go down that route anyway.  Had a similar issue a few months back and reboot did clear it but never found out  what caused it so this time wanted to try and see what it was.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I would encourage you to, make sure hosts and vCenter Server are on the latest releases.
Paul SolovyovskySenior IT AdvisorCommented:
you can always open a ticket with vmware and send them over the logs, seems like this happens from time to time.
compdigit44Commented:
Have you checked the Security Profile / Firewall settings to make sure it is not blocking connections... probably not but always worth a check
nchtIT Operations ManagerAuthor Commented:
Apologies for not getting back.  Ended up rebooting hosts to fix and awaiting VMWare response to logs.
nchtIT Operations ManagerAuthor Commented:
I've requested that this question be closed as follows:

Accepted answer: 0 points for ncht's comment #a40934630

for the following reason:

VMWare support fixed the issue.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
My first post http:#a40844803 states Shutdown VMs and Restart Hosts!
nchtIT Operations ManagerAuthor Commented:
Andrew,

Apologies, had a quick scan and did not pick up on your post.  Had done previously to fix as I said in a later post but wanted to try and isolate what was causing it. Happy to give you the points.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.