vSphere Web Client will not start after vCenter appliance restarted

We have a vCenter Server appliance ( Build 3634791).  All was running fine until we shut down the appliance (cleanly) for a planned power outage.  Since it came back up, it is no longer possible to access it either with the vCenter Web Client or the vSphere Client.  There have been no other changes made.

The vSphere Client reports:
Could Not Connect
vSphere Client could not connect to "vcenter".

An unknown connection error occurred. (The server could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable.))

The Web Client shows:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f085805ee10] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

We have confirmed that the vSphere Client service will not start:
vcenter:~ # service vsphere-client restart
Last login: Wed May 16 14:25:51 UTC 2018 on tty1
Stopping VMware vSphere Web Client...
VMware vSphere Web Client was not running.
Starting VMware vSphere Web Client...
Waiting for VMware vSphere Web Client..................
WARNING: VMware vSphere Web Client may have failed to start.

Open in new window

What steps must we take to fix this?
David HaycoxConsultant EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It can take up to 30 minutes for the Appliance to settle down, after a restart depending upon resources, disk i/o, host cpu etc..

so has this been hours ?

restart and wait 30 minutes ?

then check storage space is not full ? on the appliance

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
David HaycoxConsultant EngineerAuthor Commented:
It's been several hours, we also restarted once more already and waited.   Forgive my ignorance, how would I check the storage space?
David HaycoxConsultant EngineerAuthor Commented:
Ah yes: "The /,/storage/log filesystem is low on disk space or inodes" (from the VAMI)
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

David HaycoxConsultant EngineerAuthor Commented:
David HaycoxConsultant EngineerAuthor Commented:
Ah, great: "Access to resource settings on the host is restricted to the server that is managing it"

Chicken / egg situation: can't fix vCenter without increasing disk space, can't increase disk space without fixing vCenter.

Disassociate host from vCenter temporarily?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
You can connect direct to host ? which is hosting vCenter appliance
David HaycoxConsultant EngineerAuthor Commented:
Yes, have now increased the log drive and restarted the vCenter appliance.  Same problem, alas.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Do you have a complex vSphere environment, how many hosts and VMs ?

by complex, any of the following vDS, NSX, VDP, SRM, VR, vRops etc
David HaycoxConsultant EngineerAuthor Commented:
No, it's Essentials Plus with just three hosts and shared iSCSI storage.  Looks like /dev/sda3 is full as well:
vcenter:~ # df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/sda3                              11G   11G     0 100% /
udev                                  4.0G  164K  4.0G   1% /dev
tmpfs                                 4.0G   36K  4.0G   1% /dev/shm
/dev/sda1                             128M   41M   81M  34% /boot
/dev/mapper/core_vg-core               25G  2.5G   21G  11% /storage/core
/dev/mapper/log_vg-log                 20G   12G  7.5G  60% /storage/log
/dev/mapper/db_vg-db                  9.9G  462M  8.9G   5% /storage/db
/dev/mapper/dblog_vg-dblog            5.0G  219M  4.5G   5% /storage/dblog
/dev/mapper/seat_vg-seat              9.9G  594M  8.8G   7% /storage/seat
/dev/mapper/netdump_vg-netdump       1001M   18M  932M   2% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy  9.9G  151M  9.2G   2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc          5.0G  160M  4.6G   4% /storage/invsvc

Open in new window

So I followed this article: https://kb.vmware.com/s/article/2149278

Started services and now looking good!
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Glad it's all working again....

Yes, it happens, when disk fills up!

Regular weekly check!

It does get better with 6.5 and 6.7....
David HaycoxConsultant EngineerAuthor Commented:
Thanks, yes I recently did a 6.5 install - then 6.7 came out 3 days later!  Still, early adoption is not always a good thing.  Thanks for the pointers, I wasn't even looking at disk space.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I'd hold off 6.7 at present unless you are starting new implementation...and look at U1!!!
David HaycoxConsultant EngineerAuthor Commented:
Yes, always wait for the first service pack!  Thing is, "old" deployments of VMware just keep on working, excepting the odd issue like this one (and the one where it needs a restart if it's been up for over a year).  It's not like Windows...!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.