vCenter server 2012 error 503

We have vCenter 6.5 installed on server 2012. We planned to go to 6.7 VCSA. However, the 2012 VM restarted and now vcenter web client doesnt come back up.  We get the following:


503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:0x000000deb87e6ca0] _serverNamespace = /vsphere-client action = Allow _port = 9090)

The problem we have is that there are still hosts that have 5.5 that had not been upgraded along with some distributed switches on older version that had not been updated.  What should be the route to restore access to 6.5 so we can upgrade hosts and vDS then move to 6.7 VCSA?
LVL 2
Leigh KalbliAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Murali SripadaCommented:
Can you go to c:\program files\vmware\vcenterserver\bin  from administrator command prompt
run service-control --status --all and share us the output please
Murali SripadaCommented:
Any recent changes on the vcenter or does the issue started post vcenter reboot?
Leigh KalbliAuthor Commented:
error.PNG
No changes. Just started after what appears to be a SAN got disconnected and the VM restarted.
The 7 Worst Nightmares of a Sysadmin

Fear not! To defend your business’ IT systems we’re going to shine a light on the seven most sinister terrors that haunt sysadmins. That way you can be sure there’s nothing in your stack waiting to go bump in the night.

Murali SripadaCommented:
All major services are started except webclientsvc and vsphere-ui

can you start them manually using the below commands

service-control --start vsphere-ui
service-control --start vspherewebclientsvc
Leigh KalbliAuthor Commented:
error3.PNG
Murali SripadaCommented:
Please try the other command for webclientsvc as well.. if failed, please share us the log file (vsphere_client_virgo.log) from c:\programdata\vmware\vcenterserver\logs\vsphere-client\logs
Leigh KalbliAuthor Commented:
error4.PNG
been there for about 4 minutes.
Murali SripadaCommented:
Please share the log file which gives us some idea on what's happening..
Leigh KalbliAuthor Commented:
Murali SripadaCommented:
[2018-08-09T12:01:32.373-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-372' generated
[2018-08-09T12:01:32.374-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-373' generated
[2018-08-09T12:01:32.381-04:00] GiLogServiceListener@4409f6f  org.osgi.service.log.LogService                                   Bundle org.eclipse.equinox.ds_1.4.0.v20120112-1400, [SCR] Unexpected exception occurred! java.lang.IllegalStateException: BundleContext is no longer valid
      at org.eclipse.osgi.framework.internal.core.BundleContextImpl.checkValid(BundleContextImpl.java:931)
      at org.eclipse.osgi.framework.internal.core.BundleContextImpl.getServiceReferences(BundleContextImpl.java:498)
      at org.eclipse.equinox.internal.ds.Reference.hasProviders(Reference.java:127)
      at org.eclipse.equinox.internal.ds.Resolver.selectNewlyUnsatisfied(Resolver.java:600)
      at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:364)
      at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
      at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
      at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
      at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
      at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
      at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
      at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
      at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:225)
      at org.eclipse.equinox.internal.region.RegionManager.stop(RegionManager.java:67)
      at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:771)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:764)
      at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:510)
      at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:566)
      at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1206)
      at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:592)
      at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)
      at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
      at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:284)
      at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:692)
      at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:600)
      at org.eclipse.osgi.framework.internal.core.InternalSystemBundle$1.run(InternalSystemBundle.java:261)
      at java.lang.Thread.run(Thread.java:748)

[2018-08-09T12:01:32.385-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-384' generated
[2018-08-09T12:01:32.386-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-385' generated
[2018-08-09T12:01:32.387-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-386' generated
[2018-08-09T12:01:32.420-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-419' generated
[2018-08-09T12:01:32.421-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-420' generated
[2018-08-09T12:01:32.433-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-432' generated
[2018-08-09T12:01:32.444-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-443' generated
[2018-08-09T12:01:32.466-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-465' generated
[2018-08-09T12:01:32.478-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-477' generated
[2018-08-09T12:01:32.479-04:00] [INFO ] Thread-12                     org.eclipse.virgo.medic.eventlog.default                         ME0003I Dump 'C:\PROGRA~3\VMware\VCENTE~1\logs\VSPHER~1\dump\2018-08-09-12-01-478' generated

looks like files are corrupted or config file might be corrupted related to the webclient during reboot

If you have support entitlement, please engage vmware support
Leigh KalbliAuthor Commented:
We do.  I will reach out to them and engage a case and post back.
Leigh KalbliAuthor Commented:
case opened with VM Ware and their immediate response was to restore a back up of 5.5.  Not really acceptable in this situation considering the changes that were made since it was upgraded to 6.5. And back ups control is beyond my control so arguing about having a more current back up is a dead point.
Leigh KalbliAuthor Commented:
vmwares response was to delte the server and reinstall vcenter and all settings.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VCSA

From novice to tech pro — start learning today.