Link to home
Start Free TrialLog in
Avatar of Tom Carr
Tom CarrFlag for United Kingdom of Great Britain and Northern Ireland

asked on

503 Error connecting to vCenter Server

Hi All,

I get a 503 error when trying to login to a vCenter that we inherited.  I'm currently running VCSA 6.0.0.30700 with two hosts. I've done some investigating and noticed that several of the virtual disks were full on the VCSA, I have since extended and expanded these so I now have free space. The IP address of the vCenter is not used by anything else as it is outside of the DHCP scope on our network.

The error I see is:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f58f0069f40] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

When I run service-control --status --all I can see that the following services are not running:

VMware Component Manager)
vmware-eam (VMware ESX Agent Manager)
vmware-invsvc (VMware Inventory Service)
vmware-mbcs (VMware Message Bus Configuration Service)
vmware-netdumper (VMware vSphere ESXi Dump Collector)
vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter)
vmware-sca (VMware Service Control Agent)
vmware-vapi-endpoint (VMware vAPI Endpoint)
vmware-vdcs (VMware Content Library Service)
vmware-vpx-workflow (VMware vCenter Workflow Manager)
vmware-vpxd (VMware vCenter Server) vsphere-client ()

The services do not start if I run service-control --start --all, I need to get this up and running fairly quickly as I need to perform a vMotion. We use HP StoreVirtual VSA's for shared storage so I'm assuming I can do this.

In addition to this if I can login to vcenter:5480, after expanding the disks Overall Health changed to Good:
User generated image
Has anyone seen this before? I'm considering downloading VCSA and resetting it up from scratch, if I do this what impact will it have on our setup? Will the StoreVirtual VSA's be affected?

If anyone is able to offer advice that would be much appreciated.
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Have you restarted and waited 30 minutes ?

If this is a simple environment, re-deploy VCSA 6.5, your VCSA has broken.
Avatar of Tom Carr

ASKER

Hi Andrew,

I have restarted and waited several times. The environment is two production ESXi hosts and a third for management VMs. Storage is on HP StoreVirtual with a VSA on each of the hosts and a FOM on the management host. Will this likely be affected by the rebuild?

The time looked to be off by over an hour, I have just updated this and am now waiting for it to reboot.
A new deployed  vCenter Server should not cause you issues?

any distributed switches, NSX, VDP, SRM ?
None that I am aware of, I'm assuming I'll be able to see these on individual hosts under Configuration > Networking?
if you do not know what they are it's unlikely you have them, for a small installation.

I would discard, power off current vCenter Server, VCSA, and deploy a new one, and then re-add hosts, this will not affect any running VMs, because VCSA is just a management server.
Avatar of Murali Sripada
Murali Sripada

component manager service failed to start.. which is key here... check the space on vcsa or provide us the /var/log/vmware/cm.log and I will help you after checking the logs..
Hi Murali,

The VCSA did run out of space on two of its disks but I have since expanded these, each volume now has plenty of free space.

The CM log contains the following:

2018-08-08T16:03:03.983Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate chain is trusted
2018-08-08T16:03:03.983Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate chain is trusted
2018-08-08T16:03:04.032Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate assertion verified
2018-08-08T16:03:04.032Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate assertion verified
2018-08-08T16:03:04.032Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate assertion verified
2018-08-08T16:03:04.040Z [WrapperListener_start_runner  WARN  com.vmware.cis.services.cm.service.util.LsUtils ()] Call to lookup service failed; uri:https://10.2.0.51/lookupservice/sdk [com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404]
2018-08-08T16:03:04.041Z [WrapperListener_start_runner  WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:04.041Z [WrapperListener_start_runner  WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:04.042Z [WrapperListener_start_runner  WARN  com.vmware.cis.services.cm.service.SecurityAgent ()] Failed to connect to SSO service; e: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404, attempt 46 of 60, retry in 5s, left:69s
2018-08-08T16:03:09.050Z [WrapperListener_start_runner  INFO  com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.2.0.51/lookupservice/sdk
2018-08-08T16:03:09.057Z [WrapperListener_start_runner  INFO  com.vmware.cis.services.cm.service.util.LsUtils ()] LS URI is https://10.2.0.51/lookupservice/sdk
2018-08-08T16:03:09.094Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate chain is trusted
2018-08-08T16:03:09.094Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate chain is trusted
2018-08-08T16:03:09.094Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate chain is trusted
2018-08-08T16:03:09.140Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate assertion verified
2018-08-08T16:03:09.140Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate assertion verified
2018-08-08T16:03:09.140Z [WrapperListener_start_runner  DEBUG com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager ()] Server certificate assertion verified
2018-08-08T16:03:09.148Z [WrapperListener_start_runner  WARN  com.vmware.cis.services.cm.service.util.LsUtils ()] Call to lookup service failed; uri:https://10.2.0.51/lookupservice/sdk [com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404]
2018-08-08T16:03:09.149Z [WrapperListener_start_runner  WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:09.149Z [WrapperListener_start_runner  WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:09.150Z [WrapperListener_start_runner  WARN  com.vmware.cis.services.cm.service.SecurityAgent ()] Failed to connect to SSO service; e: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404, attempt 47 of 60, retry in 5s, left:64s
2018-08-08T16:03:04.040Z [WrapperListener_start_runner  WARN  com.vmware.cis.services.cm.service.util.LsUtils ()] Call to lookup service failed; uri:https://10.2.0.51/lookupservice/sdk [com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404]
2018-08-08T16:03:04.041Z [WrapperListener_start_runner  WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase ()] Shutting down the connection monitor.

SSO services are not responding to component manager..

can you check if the vmware-sts-idmd and vmware-stsd services are up and running. Have you rebooted the vCenter appliance?
take ssh to vcsa  
service-control --status vmware-sts-idmd and service-control --status vmware-stsd
if they both running..

check in top if any service user name is showing up as vmware-cm ..if yes kill the pid beside that.. and then stop all and start all services
also try ps -ef |grep idmd and check if they are showing 3 entries or 1

Thanks,
MS
When I run ps -ef |grep idmd I see the following:

root     15575     1  0 16:32 ?        00:00:00 vmware-sts-idmd -procname vmware
root     15576 15575  1 16:32 ?        00:00:05 vmware-sts-idmd -procname vmware
root     31821 20020  0 16:37 pts/0    00:00:00 grep idmd
ASKER CERTIFIED SOLUTION
Avatar of Murali Sripada
Murali Sripada

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Awesome! It's just come up now! Phew, many thanks for your assistance, the space issue had re-appeared after increasing the volume size, removed a couple of tar.gz files that were over 10GB and that appears to have resolved the issue!
Thanks all!