troubleshooting Question

vCenter server 6.7 - 503 Service Unavailable

Avatar of Jacob Glassman
Jacob Glassman asked on
13 Comments1 Solution474 ViewsLast Modified:
Our vCenter server doesn't seem to come up even after multiple reboots

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f7860004670] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

root@XXXX [ ~ ]# service-control --status
 applmgmt pschealth vmcam vmonapi vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsan-dps
 lwsmd vmafdd vmcad vmdird vmdnsd vmware-analytics vmware-certificatemanagement vmware-cis-license vmware-eam vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

Every log file I check seems to point to some other service which is not working but I'm not sure which core service is failing

This happened after I upgraded its ESXi host from 6.0 to 6.5.  Also around the same time our active directory domain controller was changed but my AD login works fine.

If I login to the UI then I see the below message and I can't see my ESXi servers in it :

"Could not connect to one or more vCenter Server systems:https://xxxxxxxxxxxxxxxxxxxx:443/sdk"

root@xxxxxxxx [ ~ ]# service-control --start --verbose --all
Operation not cancellable. Please wait for it to finish...
Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True
Performing start operation on service lwsmd...
2020-03-26T11:32:13.809Z  Running command: ['/usr/bin/systemctl', 'is-enabled', 'lwsmd']
2020-03-26T11:32:13.820Z  Done running command
2020-03-26T11:32:13.821Z  Running command: ['/sbin/service', 'lwsmd', 'status']
2020-03-26T11:32:13.892Z  Done running command
Successfully started service lwsmd
Performing start operation on service vmafdd...
2020-03-26T11:32:13.894Z  Running command: ['/usr/bin/systemctl', 'is-enabled', 'vmafdd']
2020-03-26T11:32:13.971Z  Done running command
2020-03-26T11:32:13.971Z  Running command: ['/sbin/service', 'vmafdd', 'status']
2020-03-26T11:32:14.039Z  Done running command
Successfully started service vmafdd
Performing start operation on service vmdird...
2020-03-26T11:32:14.041Z  Running command: ['/usr/bin/systemctl', 'is-enabled', 'vmdird']
2020-03-26T11:32:14.059Z  Done running command
2020-03-26T11:32:14.060Z  Running command: ['/sbin/service', 'vmdird', 'status']
2020-03-26T11:32:14.117Z  Done running command
Successfully started service vmdird
Performing start operation on service vmcad...
2020-03-26T11:32:14.119Z  Running command: ['/usr/bin/systemctl', 'is-enabled', 'vmcad']
2020-03-26T11:32:14.134Z  Done running command
2020-03-26T11:32:14.134Z  Running command: ['/sbin/service', 'vmcad', 'status']
2020-03-26T11:32:14.200Z  Done running command
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
2020-03-26T11:32:14.202Z  Running command: ['/usr/bin/systemctl', 'is-enabled', 'vmware-sts-idmd']
2020-03-26T11:32:14.218Z  Done running command
2020-03-26T11:32:14.219Z  Running command: ['/sbin/service', 'vmware-sts-idmd', 'status']
2020-03-26T11:32:14.276Z  Done running command
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
2020-03-26T11:32:14.278Z  Running command: ['/usr/bin/systemctl', 'is-enabled', 'vmware-stsd']
2020-03-26T11:32:14.296Z  Done running command
2020-03-26T11:32:14.296Z  Running command: ['/sbin/service', 'vmware-stsd', 'status']
2020-03-26T11:32:14.359Z  Done running command
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
2020-03-26T11:32:14.361Z  Running command: ['/usr/bin/systemctl', 'is-enabled', 'vmdnsd']
2020-03-26T11:32:14.378Z  Done running command
2020-03-26T11:32:14.378Z  Running command: ['/sbin/service', 'vmdnsd', 'status']
2020-03-26T11:32:14.444Z  Done running command
Successfully started service vmdnsd
Performing start operation on profile: ALL...
2020-03-26T11:32:14.450Z  Running command: ['/sbin/service', 'vmware-vmon', 'status']
2020-03-26T11:32:14.517Z  Done running command
2020-03-26T11:32:14.518Z  Running command: ['/usr/lib/vmware-vmon/vmon-cli', '--batchstart', 'ALL']
2020-03-26T11:44:17.553Z  Done running command
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start topologysvc, vmonapi, applmgmt, cm services. Error: Operation timed out
Jacob Glassman

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 13 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 13 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros