Solved

VCenter 5.5 service always stopping

Posted on 2013-12-19
29
7,894 Views
1 Endorsement
Last Modified: 2015-04-01
Hi,

I have a fresh install of vcenter 5.5 on Windows Server 2008 r2, I added my hosts and everything was working great.  Then I rebooted the server... now the service VMware Virtual Center Server starts on boot for about 5 seconds, then stops.

I've tried the fix here mentioned here, but it didn't work for me: https://communities.vmware.com/thread/458320 this is stopping the vcenter service, then restarting the identity management service and the secure token service, then starting the vcenter service back up.

Does anyone have any suggestions?

attached is windows event log error.

some errors I'm seeing in the vpxd log

013-12-20T18:56:36.990+13:00 [05500 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance not found, instance name not known
2013-12-20T18:56:36.990+13:00 [05500 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance not found, url not known
2013-12-20T18:56:36.990+13:00 [05500 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance not found, url not known


2013-12-20T18:56:40.926+13:00 [05500 error 'profileprofileUtil'] [Vpxd::Prof::Util::DeserializeFromFile] reading failed: FileIO error: Could not find file  : C:\ProgramData\VMware\VMware VirtualCenter\hpMetadataCache.xml


2013-12-20T18:56:40.988+13:00 [05500 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance not found, instance name not known
2013-12-20T18:56:40.988+13:00 [05500 error 'vpxdvpxdHealth'] [VpxdHealth::GetHealthEndpointInfo] Failed to get endpoint C85EFD1F-81C2-48C1-AC3B-006BA941B7E6/VIMWEBSVC

error 'win32vpxLdap_win32'] [LDAP Client] Failed to add LDAP entry cn=XXXXXXXXXXXXXXXXXXXXXXX(edited),ou=Licenses,ou=Licensing,dc=virtualcenter,dc=vmware,dc=int: 0x68 (The object already exists.)

2013-12-20T18:56:43.655+13:00 [05760 error 'licenseldapStorage'] [LicMgr] No revision store with

 [VpxdHostAccess] Failed to get VpxaCnxInfo over SOAP version vpxapi.version.version8 for host
pastedImage-0.png
1
Comment
Question by:ts11
  • 12
  • 10
  • 3
  • +3
29 Comments
 
LVL 11

Expert Comment

by:Manjunath Sullad
ID: 39731626
Please check its contacting to database server or not,

Also confirm database server is up and working fine,

Check with Database team for any errors from database side,

Also update your latest OS patches and hotfixes.

Try disabling antivirus and check (If working fine, Check with antivirus vendor on this)
0
 

Author Comment

by:ts11
ID: 39732269
Database seems to be fine.  It's installed on the same server. All running and no errors.

The server is fully patched.

There's no antivirus currently installed on the machine.
0
 
LVL 8

Expert Comment

by:Jeff Perry
ID: 39732922
I had a similar issue with 5.5 our solution after working with VMware support was to make sure the same user account was starting the following services.

VMware VirtualCenter Management Webservices
VMware VirtualCenter Server
VMware vSphere Profile-Driven Storage Service
VMware vSphere Upate Manager Service

This user account was also the user given ownership of the databases on the vCenter Server.
0
 

Author Comment

by:ts11
ID: 39734109
I updated those 4 services with an enterprise admin account.  I also deleted the duplicate LDAP entry as mentioned on this article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2044680

Still the issue persists.  I'm not sure why but even after removing the entry and starting the vcenter service, the vpxd log still says there is a duplicate of that same entry (which I confirmed is back).
0
 

Author Comment

by:ts11
ID: 39735605
Tried setting the service to Disabled, then rebooting, then deleting the entry in ADSI edit, but once I turn on the service it fails and I still get this error for some reason:

error 'win32vpxLdap_win32'] [LDAP Client] Failed to add LDAP entry cn=XXXXXXXXXXXXXXXXXXXXXXX(edited),ou=Licenses,ou=Licensing,dc=virtualcenter,dc=vmware,dc=int: 0x68 (The object already exists.)

Does anyone know anything I can try?
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39736005
If your database is on the same server, then I would expect to see a connection error after you reboot because the sql express services are not yet started before the vcenter service attempts to contact it.  You can make the vcenter service dependent on the sql services to solve that, but really once you wait after a reboot and restart that service it should stay up just fine.

As for the duplicate issue, are you stopping the WWW service too when deleting?  Does this issue cause any problems - not sure you are clear whether this is causing issues or just polluting the logs.
0
 

Author Comment

by:ts11
ID: 39736667
Checked for SQL errors but haven't found any.

WWW service isn't installed on this server, it's purely for Vcenter.

I thought it was causing my issues due to the vmware article on it I posted above.  I also think it started happening after I added an identity source to vcenter (my domain), which was working fine till the reboot.
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39736695
This version of vCenter uses a web admin portal as well as allowing the client, so have a look to see if it installed itself during the installation on that server.  Since I see this "VMware VirtualCenter Management Webservices", I imagine it is.  I doubt the ldap error is causing your problem, and it only sounds as if you are seeing the service failure after a reboot - which I would expect.

You can go into the properties of the vCenter service and add the dependency for the SQL service which should make it wait until SQL is up before it tries to start.

Every error message you posted tells me the DB is not running when the VCenter Management Service tries to access the database.  It should, however, start after 5-10 minutes (long enough for SQL to start and normalize) and stay running.

Other than that, is there any other issues you are experiencing once the vCenter service is started?
0
 

Author Comment

by:ts11
ID: 39736796
Web admin portal is definitely installed, I was using it fine before this issue.

I've added the dependency (see screenshot). Then rebooted the server, but same issue.

The only other issue I'm seeing is the time is always out of sync after a reboot even after syncing it to an external source.
services.PNG
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39736825
Time could be an issue, certainly.  Since your sign on now is linked to AD if the time is off with respect to AD then problems will arise.

Point the ESX hosts to your DC for time and restart their time service.  As long as your VMs have the VMware tools installed, then they sync with the host.

You may need to put the dependency on the other services for VMware as well.
0
 

Author Comment

by:ts11
ID: 39736877
Well this is strange, I can't seem to keep it synced to the correct time.  All my other servers sync up fine.

I've also tried syncing to an external time source doing this: w32tm /config /manualpeerlist:"0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org" /syncfromflags:manual /reliable:yes /update
net stop w32time && net start w32time

and internal using net time /domain:mydomain /set /yes

After reboot they always go back to 30+ minutes out.

Edit: also confirmed the time source is consistent after a reboot in the registry.  Attached file.

Could this be something a vmware service on this server is doing?
time.PNG
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39737266
If your vcenter server is domain-joined, it should sync with the PDCE to avoid offset issues in AD that affect Kerberos.  This will happen if the latter is true (ESX syncs with DC).

You also need to install the VMware tools on that VM and make sure the Host ESXi server has the proper ntp settings using your DC(s) as a time source.

If the Host is wrong, the VMs will be too.
0
 

Author Comment

by:ts11
ID: 39737288
I've managed to fix the time issue by adding some entries to the vmx of this particular virtual machine.

Unfortunately doesn't seem to have resolved the issue with vcenter though.
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39737290
So, now...your saying the vcenter service still won't stay up?
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 

Author Comment

by:ts11
ID: 39737294
Yeah still the same issue.  Start the vcenter service and it crashes around 10 seconds after.

I've attached another windows error i've found in the event log.

Thanks for the help btw!
windows-error.PNG
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39737303
On the vcenter server, open up a web browser.

Go here.

https://localhost/mob/?moid=ServiceDirectory

Screenshot that and attach it for me.
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39737305
Oops, you may need to go here first and log in.

https://localhost/mob

Then paste the above URL in and go.
0
 

Accepted Solution

by:
ts11 earned 0 total points
ID: 39737383
Hmm I went to https://localhost/mob but it doesn't go any where? To a page cannot be found.

I believe I may have found a VMware KB which may be my issue, though.  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2065630

I do know we have some vms with mass amounts of snapshots that got created many times (10+) in error.

The work around is painful, but I guess this is my next step...
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39737958
Do you show those entries in the log that are referred to in that article?
0
 

Author Comment

by:ts11
ID: 39738565
Yes I have a whole bunch of these.

2013-12-20T18:56:46.650+13:00 [05796 warning 'utilvpxdDbLoad'] [ORM] Load: Missing mapping for property hardware.device.backing.backingObjectId
2013-12-20T18:56:46.650+13:00 [05796 warning 'utilvpxdDbLoad'] [ORM] Load: Missing mapping for property hardware.device.backing.backingObjectId
2013-12-20T18:56:46.664+13:00 [05796 info 'Default'] CoreDump: Writing minidump
2013-12-20T18:56:47.536+13:00 [05988 warning 'VpxProfiler' opID=SWI-41a7] VpxUtil_InvokeWithOpId [TotalTime] took 12237 ms

This is right at the end of the log.

Although, I don't see why I need to delete the vms... couldn't I just log into the hosts themselves and remove the snapshots from the vms that have too many?
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39738606
They are getting you to purge the db of bad entries.  Your virtual machines can be re-added to inventory from the datastore afterwards.
0
 

Author Comment

by:ts11
ID: 39738796
If I purge the db of bad entries for a vm and that vm is running, once I turn on the vcenter (which my hosts are attached to) what impact would that have on the vm?
0
 
LVL 51

Expert Comment

by:Netman66
ID: 39738812
It shouldn't impact the running vm.

Test it if you wish.
0
 

Author Closing Comment

by:ts11
ID: 39829103
Link provided shows the solution to my issue.
0
 

Expert Comment

by:TPCSGP
ID: 40655667
Hi, May i know what was the solution to this problem?
0
 

Expert Comment

by:erehnquist
ID: 40688528
Please provide an update to the solution.  I have two new Win 2012 vCenter 5.5 U2 servers using SQL 2012,
and after installing the pieces of vCenter in the right order, either with Simple Install or one by one,
vCenter works fine until reboot.  After reboot, I cannot get the virtualCenter Server service started.
I verified that IIS and World Wide Publishing are not installed.  I also followed VMware KB 1003926 (a step by step) and 1003971 (ports in use), but to no avail.  Any help is appreciated.

Thanks!
0
 

Expert Comment

by:erehnquist
ID: 40701184
I found a solution that works for us.  Although I tried every KB, Communities, and Blog post I could find, nothing would allow vCenter services to start after the first reboot after installation.  I reinstalled several times, and like usual, it works until reboot.  I used Netstat to see what PID was using port 80 (always PID #4), but it only revealed that the NT Kernel was using it.  I reinstalled again, only this time, I changed the port from 80 to 81 and vCenter Service started after reboot!  I then remembered that, during the previous installations, the installation was stopped as the installer warned of a conflict with port 80.  KB’s said it was probably SQL Server Reporting Services, and with each install, I would stop that service and the vCenter installation would continue.  

So, I reinstalled again, stopping SQL Server Reporting Service as usual, keeping vCenter on port 80, rebooted and again, and as expected, vCenter Services would not start.  I then stopped SQL Server Reporting Services after that reboot and sure enough, vCenter Services would manually start.  

My permanent  solution then was to make the registry hack so SQL Server Reporting Service is now dependent on vCenter Service (vpxd) starting up first.  To do that, I made a new Multi-String Value called “DependentOnService” (without the quotes) to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ReportServer and set the Data Value to “vpxd” (without the quotes).  My order of dependencies are now:

1. SQL Server Service (by default)
2. vCenter Service (by default: delayed start)
3. SQL Server Reporting Services (dependent on vpxd)

I hope this helps somebody else!  It was painful for me!
0
 

Expert Comment

by:erehnquist
ID: 40701255
OK, last comment, I just realized that making SSRS dependent on vpxd will grant vCenter first use to port 80.  That will more than likely keep SSRS from working properly, thought the service still starts.  If SSRS is needed, you will need to find a different port for it or vCenter.  Again, this assumes running SQL and vCenter on the same server.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

In this article, I will show you HOW TO: Suppress Configuration Issues and Warnings Alert displayed in Summary status for ESXi 6.5 after enabling SSH or ESXi Shell.
In this article, I will show you HOW TO: Install VMware Tools for Windows on a VMware Windows virtual machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, using the VMware Host Client. The virtual machine has Windows Server 2016 instal…
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…
Advanced tutorial on how to run the esxtop command to capture a batch file in csv format in order to export the file and use it for performance analysis. He demonstrates how to download the file using a vSphere web client (or vSphere client) and exp…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now