Solved

Unable to connect to the MKS in vSphere are rebooting ESXi 4.1 host

Posted on 2013-06-08
16
1,577 Views
Last Modified: 2013-07-07
After rebooting an ESXI host, I'm not longer able to view the console sessions in the Console tabs!

What I get is a message bar at the top that states:

"Unable to connect to the MKS:  Failed to connect to the server XX.XX.XX.XX:902"

I've Googled this that all I can find is suggestions on modifying the host file and modifying firewall settings, but I can replicate on the same network segment and on different laptops with different users.

There are a number of virtual servers on this host with all but one working as expected outside of this, except one, which is the file server, has stopped publishing network shares, which it gets from ISCSI mappings via the ESXI host.

On typing this, there may be one thing that I have changed since the reboot, which oddly enough was to disconnect the physical KVM from it.  I find it hard to believe this could be the culprit, but I thought I'd add this point just in case someone else has come up against this issue and MKS does stand for Mouse, Keyboard, Screen after all!!  By the way, I'm working remote form it at the moment, but can go in if need be.

Urgent assistance would really be appreciated if anyone picks this one up.

Kind Regards

Lee
0
Comment
Question by:FSIFM
  • 9
  • 7
16 Comments
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
This can be a number of reasons, access to port 902.

1. any firewalls changed on your network?

2. does this happen to all Virtual Machines?

3. can you telnet to <ip address of host> 902

4. can you telnet to <dns or hostname of host> 902

5. repeat 3 & 4 above with ports 80 and 443

6. can you ping by IP Address and hostname

7. can you traceroute by IP address and hostname e.g. reverse DNS

8. there is no router, internet or NAT involved? e.g. is this local LAN

9. Restart Network Management Agents on Host

10. try restarting

11. un-register one of the VMs, and re-register with inventory.
0
 
LVL 4

Author Comment

by:FSIFM
Comment Utility
1. any firewalls changed on your network?  
No, but despite them being rebooted, giving the possibility of a un-saved config, I can replicate this on the same network segment with no firewalls in between.

2. does this happen to all Virtual Machines?  
Only on this esxi host yes.  Another 4.1.0 esxi host does not have this issue after being rebooted.

3. can you telnet to <ip address of host> 902
No

4. can you telnet to <dns or hostname of host> 902
No

5. repeat 3 & 4 above with ports 80 and 443
Yes I can telnet on these ports.

6. can you ping by IP Address and hostname
Yes

7. can you traceroute by IP address and hostname e.g. reverse DNS
Yes

8. there is no router, internet or NAT involved? e.g. is this local LAN
Yes its local LAN

9. Restart Network Management Agents on Host
Done, but no luck.

10. try restarting
Done many times. 8-(

11. un-register one of the VMs, and re-register with inventory.
No joy on this one either.
 
I'm in the office now and have connected the physical KVM and rebooted just in case there is an odd bug, but no luck on that one either.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
You should get a connection to 902...

something like the following should respond:-

220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtoco
l:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported

tried a /sbin/services.sh restart from the console or ssh
0
 
LVL 4

Author Comment

by:FSIFM
Comment Utility
I'm getting a connection refused when I try to create a SSH connection (via Putty), but I can confirm that Remote Tech Support (SSH) service is running (via the vSphere Client).

From the vSphere client, I have manually restarted the services, but that hasn't helped either.  is this that same thing as running this command?

I can telnet on port 902 to another 4.1 esxi host, so its something on this particular host that blocking it as its all on the same network segment of our LAN.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
firewalls will block tcp 902

or, ESXi 4.1 host is broke.

no, services.sh, restarts ALL the services.
0
 
LVL 4

Author Comment

by:FSIFM
Comment Utility
Definitely no firewalls in between.  I can replicate from a server (again on same network segment) that doesn't have a local firewall on.

ESXi host being broke is looking like a possibility, is there a way to confirm this.

As I cant connect via putty, I've pressed ALT + F1 at the console login screen to enter the hidden Console, but the command doesn't run from the root.  Can you remind me the syntax that I need to run to get to the folder that the command can run from?
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
can you login? and get a console prompt?

 /sbin/services.sh restart

the above does not work?

login as: root
Using keyboard-interactive authentication.
Password:
The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ # cd /sbin
/sbin # ls services.sh
services.sh
/sbin # services.sh
Usage: services.sh {start|stop|restart}
/sbin #

Open in new window

0
 
LVL 4

Author Comment

by:FSIFM
Comment Utility
Oops, must have been a typo, yes it does work from the console prompt, but its still not fixed it.
0
VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

 
LVL 4

Author Comment

by:FSIFM
Comment Utility
Thanks for you advice so far Hanccocka.

With the exception of rebuilding the host, can you think of any thing else?  I do have a server scheduled to replace it, but ideally, the replacement cant happen until 26th June.

Another thing that may be relevant is that we have a Dell MD 3000i hanging off the host as well (via 2 virtual switches), and one virtual server that utilises it.  Out of all the virtual servers on the host, this seems to the only one that doesn't work!!  Perhaps a little more serious than the ESXi issue, but I'm hoping the issue is related, hence I'm focussing on ESXi first.

I've checked that the iSCSI cables are in the correct VLANs etc., so if you have any further ideas, your input will be appreciated.
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
Comment Utility
something is wrong with services, it's quicker to re-install (only takes 7mins).

and after re-install, it should not take long to re-configure 30mins.
0
 
LVL 4

Author Comment

by:FSIFM
Comment Utility
I was thinking of doing an in place upgrade to 5.1.  Can you see any issues when that?

As a precaution, I'm trying to copy some of the key servers from the datastore on to my local disk, but I'm getting "Failed to log into nfc server" error.  According to a post I found (http://communities.vmware.com/message/857754#857754), this also suggests an issue with port 902!  The suggestions in this didn't help either. 8-(  Haven't tried connecting directly to the ESXi server yet thought, but that's not exactly a fix.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
If I had an issue with a server, I would not do an in-place upgrade, you could carry the issues forward.

HOW TO: Connect to the VMware vSphere Hypervisor 5.1 (ESXi 5.1) using the vSphere Client

Re-installation can maintain the VMs on the VMFS datastore, if selected.

Also check your server is supported and on the HCL, requirements have changed for 5.1.

Check the VMware HCL (Hardware Compatability Lists) here

http://www.vmware.com/go/hcl

http://www.vmware.com/resources/compatibility/search.php
0
 
LVL 4

Author Comment

by:FSIFM
Comment Utility
Bit worried now!  There's a few virtual servers on the System drive, so I tested moving a few ISOs between this and any of the other 2 local datastores and it comes back saying "Error caused by file   /vmfs/volumes/..."

If I boot from a 5.1 ISO and install over the top, will it format the system drive?  I cant afford to lose the virtual servers that reside on this drive.
0
 
LVL 117

Expert Comment

by:Andrew Hancock (VMware vExpert / EE MVE)
Comment Utility
No, it will ask the question preserve VMFS (e.g. VMs on dataastore), select Yes.
0
 
LVL 4

Author Comment

by:FSIFM
Comment Utility
Just to keep you posted, I've got the ESXi 5.1 media ready, but need to plan the install as its got a number of production servers on this box which mean I need to coordinate some downtime.
0
 
LVL 4

Author Closing Comment

by:FSIFM
Comment Utility
I didn't exactly re-install, but instead I installed ESXi 5.1 to a USB stick on an identical server and booted from it on the problematic server.  This picked up the datastores and I was able to copy files to and from them.  I could also view the console as expected without getting the MKS error.
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

One of the new features of a version 7.0 or later virtual machine, supported in VMware vSphere 4.1, 5.0 or the VMware vSphere Hypervisor ESXi 4.1, ESXi 5.0 often overlooked by VMware Administrators is the ability to add and connect USB devices conne…
David Varnum recently wrote up his impressions of PRTG, based on a presentation by my colleague Christian at Tech Field Day at VMworld in Barcelona. Thanks David, for your detailed and honest evaluation!
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
Teach the user how to configure vSphere clusters to support the VMware FT feature Open vSphere Web Client: Verify vSphere HA is enabled: Verify netowrking for vMotion and FT Logging is in place or create it: Turn On FT for a virtual machine: Verify …

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now