Tom Carr
asked on
503 Error connecting to vCenter Server
Hi All,
I get a 503 error when trying to login to a vCenter that we inherited. I'm currently running VCSA 6.0.0.30700 with two hosts. I've done some investigating and noticed that several of the virtual disks were full on the VCSA, I have since extended and expanded these so I now have free space. The IP address of the vCenter is not used by anything else as it is outside of the DHCP scope on our network.
The error I see is:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipe ServiceSpe cE:0x7f58f 0069f40] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webs erver-pipe )
When I run service-control --status --all I can see that the following services are not running:
VMware Component Manager)
vmware-eam (VMware ESX Agent Manager)
vmware-invsvc (VMware Inventory Service)
vmware-mbcs (VMware Message Bus Configuration Service)
vmware-netdumper (VMware vSphere ESXi Dump Collector)
vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter)
vmware-sca (VMware Service Control Agent)
vmware-vapi-endpoint (VMware vAPI Endpoint)
vmware-vdcs (VMware Content Library Service)
vmware-vpx-workflow (VMware vCenter Workflow Manager)
vmware-vpxd (VMware vCenter Server) vsphere-client ()
The services do not start if I run service-control --start --all, I need to get this up and running fairly quickly as I need to perform a vMotion. We use HP StoreVirtual VSA's for shared storage so I'm assuming I can do this.
In addition to this if I can login to vcenter:5480, after expanding the disks Overall Health changed to Good:
Has anyone seen this before? I'm considering downloading VCSA and resetting it up from scratch, if I do this what impact will it have on our setup? Will the StoreVirtual VSA's be affected?
If anyone is able to offer advice that would be much appreciated.
I get a 503 error when trying to login to a vCenter that we inherited. I'm currently running VCSA 6.0.0.30700 with two hosts. I've done some investigating and noticed that several of the virtual disks were full on the VCSA, I have since extended and expanded these so I now have free space. The IP address of the vCenter is not used by anything else as it is outside of the DHCP scope on our network.
The error I see is:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipe
When I run service-control --status --all I can see that the following services are not running:
VMware Component Manager)
vmware-eam (VMware ESX Agent Manager)
vmware-invsvc (VMware Inventory Service)
vmware-mbcs (VMware Message Bus Configuration Service)
vmware-netdumper (VMware vSphere ESXi Dump Collector)
vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter)
vmware-sca (VMware Service Control Agent)
vmware-vapi-endpoint (VMware vAPI Endpoint)
vmware-vdcs (VMware Content Library Service)
vmware-vpx-workflow (VMware vCenter Workflow Manager)
vmware-vpxd (VMware vCenter Server) vsphere-client ()
The services do not start if I run service-control --start --all, I need to get this up and running fairly quickly as I need to perform a vMotion. We use HP StoreVirtual VSA's for shared storage so I'm assuming I can do this.
In addition to this if I can login to vcenter:5480, after expanding the disks Overall Health changed to Good:
Has anyone seen this before? I'm considering downloading VCSA and resetting it up from scratch, if I do this what impact will it have on our setup? Will the StoreVirtual VSA's be affected?
If anyone is able to offer advice that would be much appreciated.
ASKER
Hi Andrew,
I have restarted and waited several times. The environment is two production ESXi hosts and a third for management VMs. Storage is on HP StoreVirtual with a VSA on each of the hosts and a FOM on the management host. Will this likely be affected by the rebuild?
The time looked to be off by over an hour, I have just updated this and am now waiting for it to reboot.
I have restarted and waited several times. The environment is two production ESXi hosts and a third for management VMs. Storage is on HP StoreVirtual with a VSA on each of the hosts and a FOM on the management host. Will this likely be affected by the rebuild?
The time looked to be off by over an hour, I have just updated this and am now waiting for it to reboot.
A new deployed vCenter Server should not cause you issues?
any distributed switches, NSX, VDP, SRM ?
any distributed switches, NSX, VDP, SRM ?
ASKER
None that I am aware of, I'm assuming I'll be able to see these on individual hosts under Configuration > Networking?
if you do not know what they are it's unlikely you have them, for a small installation.
I would discard, power off current vCenter Server, VCSA, and deploy a new one, and then re-add hosts, this will not affect any running VMs, because VCSA is just a management server.
I would discard, power off current vCenter Server, VCSA, and deploy a new one, and then re-add hosts, this will not affect any running VMs, because VCSA is just a management server.
component manager service failed to start.. which is key here... check the space on vcsa or provide us the /var/log/vmware/cm.log and I will help you after checking the logs..
ASKER
Hi Murali,
The VCSA did run out of space on two of its disks but I have since expanded these, each volume now has plenty of free space.
The CM log contains the following:
2018-08-08T16:03:03.983Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate chain is trusted
2018-08-08T16:03:03.983Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate chain is trusted
2018-08-08T16:03:04.032Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate assertion verified
2018-08-08T16:03:04.032Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate assertion verified
2018-08-08T16:03:04.032Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate assertion verified
2018-08-08T16:03:04.040Z [WrapperListener_start_run ner WARN com.vmware.cis.services.cm .service.u til.LsUtil s ()] Call to lookup service failed; uri:https://10.2.0.51/lookupservice/sdk [com.vmware.vim.vmomi.clie nt.common. Unexpected StatusCode Exception: Unexpected status code: 404]
2018-08-08T16:03:04.041Z [WrapperListener_start_run ner WARN com.vmware.vim.vmomi.clien t.http.imp l.HttpConf igurationC ompilerBas e$Connecti onMonitorT hreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:04.041Z [WrapperListener_start_run ner WARN com.vmware.vim.vmomi.clien t.http.imp l.HttpConf igurationC ompilerBas e$Connecti onMonitorT hreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:04.042Z [WrapperListener_start_run ner WARN com.vmware.cis.services.cm .service.S ecurityAge nt ()] Failed to connect to SSO service; e: com.vmware.vim.vmomi.clien t.common.U nexpectedS tatusCodeE xception: Unexpected status code: 404, attempt 46 of 60, retry in 5s, left:69s
2018-08-08T16:03:09.050Z [WrapperListener_start_run ner INFO com.vmware.cis.services.cm .service.u til.LsUtil s ()] LS URI is https://10.2.0.51/lookupservice/sdk
2018-08-08T16:03:09.057Z [WrapperListener_start_run ner INFO com.vmware.cis.services.cm .service.u til.LsUtil s ()] LS URI is https://10.2.0.51/lookupservice/sdk
2018-08-08T16:03:09.094Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate chain is trusted
2018-08-08T16:03:09.094Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate chain is trusted
2018-08-08T16:03:09.094Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate chain is trusted
2018-08-08T16:03:09.140Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate assertion verified
2018-08-08T16:03:09.140Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate assertion verified
2018-08-08T16:03:09.140Z [WrapperListener_start_run ner DEBUG com.vmware.vim.vmomi.clien t.http.imp l.Thumbpri ntTrustMan ager ()] Server certificate assertion verified
2018-08-08T16:03:09.148Z [WrapperListener_start_run ner WARN com.vmware.cis.services.cm .service.u til.LsUtil s ()] Call to lookup service failed; uri:https://10.2.0.51/lookupservice/sdk [com.vmware.vim.vmomi.clie nt.common. Unexpected StatusCode Exception: Unexpected status code: 404]
2018-08-08T16:03:09.149Z [WrapperListener_start_run ner WARN com.vmware.vim.vmomi.clien t.http.imp l.HttpConf igurationC ompilerBas e$Connecti onMonitorT hreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:09.149Z [WrapperListener_start_run ner WARN com.vmware.vim.vmomi.clien t.http.imp l.HttpConf igurationC ompilerBas e$Connecti onMonitorT hreadBase ()] Shutting down the connection monitor.
2018-08-08T16:03:09.150Z [WrapperListener_start_run ner WARN com.vmware.cis.services.cm .service.S ecurityAge nt ()] Failed to connect to SSO service; e: com.vmware.vim.vmomi.clien t.common.U nexpectedS tatusCodeE xception: Unexpected status code: 404, attempt 47 of 60, retry in 5s, left:64s
The VCSA did run out of space on two of its disks but I have since expanded these, each volume now has plenty of free space.
The CM log contains the following:
2018-08-08T16:03:03.983Z [WrapperListener_start_run
2018-08-08T16:03:03.983Z [WrapperListener_start_run
2018-08-08T16:03:04.032Z [WrapperListener_start_run
2018-08-08T16:03:04.032Z [WrapperListener_start_run
2018-08-08T16:03:04.032Z [WrapperListener_start_run
2018-08-08T16:03:04.040Z [WrapperListener_start_run
2018-08-08T16:03:04.041Z [WrapperListener_start_run
2018-08-08T16:03:04.041Z [WrapperListener_start_run
2018-08-08T16:03:04.042Z [WrapperListener_start_run
2018-08-08T16:03:09.050Z [WrapperListener_start_run
2018-08-08T16:03:09.057Z [WrapperListener_start_run
2018-08-08T16:03:09.094Z [WrapperListener_start_run
2018-08-08T16:03:09.094Z [WrapperListener_start_run
2018-08-08T16:03:09.094Z [WrapperListener_start_run
2018-08-08T16:03:09.140Z [WrapperListener_start_run
2018-08-08T16:03:09.140Z [WrapperListener_start_run
2018-08-08T16:03:09.140Z [WrapperListener_start_run
2018-08-08T16:03:09.148Z [WrapperListener_start_run
2018-08-08T16:03:09.149Z [WrapperListener_start_run
2018-08-08T16:03:09.149Z [WrapperListener_start_run
2018-08-08T16:03:09.150Z [WrapperListener_start_run
2018-08-08T16:03:04.040Z [WrapperListener_start_run ner WARN com.vmware.cis.services.cm .service.u til.LsUtil s ()] Call to lookup service failed; uri:https://10.2.0.51/lookupservice/sdk [com.vmware.vim.vmomi.clie nt.common. Unexpected StatusCode Exception: Unexpected status code: 404]
2018-08-08T16:03:04.041Z [WrapperListener_start_run ner WARN com.vmware.vim.vmomi.clien t.http.imp l.HttpConf igurationC ompilerBas e$Connecti onMonitorT hreadBase ()] Shutting down the connection monitor.
SSO services are not responding to component manager..
can you check if the vmware-sts-idmd and vmware-stsd services are up and running. Have you rebooted the vCenter appliance?
2018-08-08T16:03:04.041Z [WrapperListener_start_run
SSO services are not responding to component manager..
can you check if the vmware-sts-idmd and vmware-stsd services are up and running. Have you rebooted the vCenter appliance?
take ssh to vcsa
service-control --status vmware-sts-idmd and service-control --status vmware-stsd
if they both running..
check in top if any service user name is showing up as vmware-cm ..if yes kill the pid beside that.. and then stop all and start all services
service-control --status vmware-sts-idmd and service-control --status vmware-stsd
if they both running..
check in top if any service user name is showing up as vmware-cm ..if yes kill the pid beside that.. and then stop all and start all services
also try ps -ef |grep idmd and check if they are showing 3 entries or 1
Thanks,
MS
Thanks,
MS
ASKER
When I run ps -ef |grep idmd I see the following:
root 15575 1 0 16:32 ? 00:00:00 vmware-sts-idmd -procname vmware
root 15576 15575 1 16:32 ? 00:00:05 vmware-sts-idmd -procname vmware
root 31821 20020 0 16:37 pts/0 00:00:00 grep idmd
root 15575 1 0 16:32 ? 00:00:00 vmware-sts-idmd -procname vmware
root 15576 15575 1 16:32 ? 00:00:05 vmware-sts-idmd -procname vmware
root 31821 20020 0 16:37 pts/0 00:00:00 grep idmd
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Awesome! It's just come up now! Phew, many thanks for your assistance, the space issue had re-appeared after increasing the volume size, removed a couple of tar.gz files that were over 10GB and that appears to have resolved the issue!
ASKER
Thanks all!
If this is a simple environment, re-deploy VCSA 6.5, your VCSA has broken.