asked on

VPXD.exe Fails to start

Hey All,

The other day my VCenter Server failed to start, so I am left unable to centrally manage my infastructure.

I am fairly new to vmware in general.

Can anyone help me out.

Below is the log i obtained by trying to run vpxd.exe manually with the -s switch.

D:\Program Files\VMware\Infrastructure\VirtualCenter Server>vpxd
[VpxdOsLayer] Couldn't read registry path SOFTWARE\VMware, Inc.\VMware VirtualCe
nter
[VpxdOsLayer] Couldn't write key DBRepairMode to registry
ASSERT d:/build/ob/bora-345043/bora/vpx/vpxd/vpxdMain.cpp:225

D:\Program Files\VMware\Infrastructure\VirtualCenter Server>vpxd.exe -s
[VpxdOsLayer] Couldn't read registry path SOFTWARE\VMware, Inc.\VMware VirtualCe
nter
[VpxdOsLayer] Couldn't write key DBRepairMode to registry
ASSERT d:/build/ob/bora-345043/bora/vpx/vpxd/vpxdMain.cpp:225
[2011-05-04 11:25:13.221 04608 info 'App'] Current working directory: D:\Program
Files\VMware\Infrastructure\VirtualCenter Server
[2011-05-04 11:25:13.227 04608 info 'App'] Log path: C:\ProgramData\VMware\VMwar
e VirtualCenter\Logs
[2011-05-04 11:25:13.229 04608 info 'App'] Initializing SSL
[2011-05-04 11:25:13.229 04608 info 'Libs'] Using system libcrypto, version 9080
EF
[2011-05-04 11:25:14.271 04608 info 'App'] Vmacore::InitSSL: doVersionCheck = tr
ue, handshakeTimeoutUs = 120000000
[2011-05-04 11:25:14.273 04608 warning 'Libs'] W32Util_RegisterEventLog: Could n
ot open registry key. Error 0.
[2011-05-04 11:25:14.274 04608 warning 'Libs'] Failed to initialize event source
for VMware VirtualCenter Server.
[2011-05-04 11:25:21.361 04608 info 'App'] CoreDump: Writing minidump
[2011-05-04 11:25:22.274 04608 panic 'App']

Panic: Failed to do early initialization.
Backtrace:
backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFil
eWriter
backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQui
ckBacktrace
backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
backtrace[06] rip 000000013fc95793 (no symbol)
backtrace[07] rip 000000013f2abc58 (no symbol)
backtrace[08] rip 000000013f2ac0c4 (no symbol)
backtrace[09] rip 000000013f2aadf2 (no symbol)
backtrace[10] rip 0000000077aff56d BaseThreadInitThunk
backtrace[11] rip 0000000077c32cc1 RtlUserThreadStart

[2011-05-04 11:25:22.345 04608 panic 'App'] Failed to do early initialization.

------ In-memory logs start --------
[2011-05-04 11:25:13.221 04608 info 'App'] Current working directory: D:\Program
Files\VMware\Infrastructure\VirtualCenter Server
[2011-05-04 11:25:13.222 04608 verbose 'ThreadPool'] TaskMax=90, IoMin=1, IoMax=
181
[2011-05-04 11:25:13.227 04608 info 'App'] Log path: C:\ProgramData\VMware\VMwar
e VirtualCenter\Logs
[2011-05-04 11:25:13.229 04608 info 'App'] Initializing SSL
[2011-05-04 11:25:13.229 04608 info 'Libs'] Using system libcrypto, version 9080
EF
[2011-05-04 11:25:14.271 04608 info 'App'] Vmacore::InitSSL: doVersionCheck = tr
ue, handshakeTimeoutUs = 120000000
[2011-05-04 11:25:14.273 04608 warning 'Libs'] W32Util_RegisterEventLog: Could n
ot open registry key. Error 0.
[2011-05-04 11:25:14.274 04608 warning 'Libs'] Failed to initialize event source
for VMware VirtualCenter Server.
[2011-05-04 11:25:21.361 04608 info 'App'] CoreDump: Writing minidump
[2011-05-04 11:25:22.274 04608 panic 'App']

Panic: Failed to do early initialization.
Backtrace:
backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFil
eWriter
backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQui
ckBacktrace
backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
backtrace[06] rip 000000013fc95793 (no symbol)
backtrace[07] rip 000000013f2abc58 (no symbol)
backtrace[08] rip 000000013f2ac0c4 (no symbol)
backtrace[09] rip 000000013f2aadf2 (no symbol)
backtrace[10] rip 0000000077aff56d BaseThreadInitThunk
backtrace[11] rip 0000000077c32cc1 RtlUserThreadStart

[2011-05-04 11:25:22.345 04608 panic 'App'] Failed to do early initialization.

------ In-memory logs end --------

D:\Program Files\VMware\Infrastructure\VirtualCenter Server>
D:\Program Files\VMware\Infrastructure\VirtualCenter Server>vpxd
[VpxdOsLayer] Couldn't read registry path SOFTWARE\VMware, Inc.\VMware VirtualCe
nter
[VpxdOsLayer] Couldn't write key DBRepairMode to registry
ASSERT d:/build/ob/bora-345043/bora/vpx/vpxd/vpxdMain.cpp:225

D:\Program Files\VMware\Infrastructure\VirtualCenter Server>vpxd.exe -s
[VpxdOsLayer] Couldn't read registry path SOFTWARE\VMware, Inc.\VMware VirtualCe
nter
[VpxdOsLayer] Couldn't write key DBRepairMode to registry
ASSERT d:/build/ob/bora-345043/bora/vpx/vpxd/vpxdMain.cpp:225
[2011-05-04 11:25:13.221 04608 info 'App'] Current working directory: D:\Program
Files\VMware\Infrastructure\VirtualCenter Server
[2011-05-04 11:25:13.227 04608 info 'App'] Log path: C:\ProgramData\VMware\VMwar
e VirtualCenter\Logs
[2011-05-04 11:25:13.229 04608 info 'App'] Initializing SSL
[2011-05-04 11:25:13.229 04608 info 'Libs'] Using system libcrypto, version 9080
EF
[2011-05-04 11:25:14.271 04608 info 'App'] Vmacore::InitSSL: doVersionCheck = tr
ue, handshakeTimeoutUs = 120000000
[2011-05-04 11:25:14.273 04608 warning 'Libs'] W32Util_RegisterEventLog: Could n
ot open registry key. Error 0.
[2011-05-04 11:25:14.274 04608 warning 'Libs'] Failed to initialize event source
for VMware VirtualCenter Server.
[2011-05-04 11:25:21.361 04608 info 'App'] CoreDump: Writing minidump
[2011-05-04 11:25:22.274 04608 panic 'App']

Panic: Failed to do early initialization.
Backtrace:
backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFil
eWriter
backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQui
ckBacktrace
backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
backtrace[06] rip 000000013fc95793 (no symbol)
backtrace[07] rip 000000013f2abc58 (no symbol)
backtrace[08] rip 000000013f2ac0c4 (no symbol)
backtrace[09] rip 000000013f2aadf2 (no symbol)
backtrace[10] rip 0000000077aff56d BaseThreadInitThunk
backtrace[11] rip 0000000077c32cc1 RtlUserThreadStart

[2011-05-04 11:25:22.345 04608 panic 'App'] Failed to do early initialization.

------ In-memory logs start --------
[2011-05-04 11:25:13.221 04608 info 'App'] Current working directory: D:\Program
Files\VMware\Infrastructure\VirtualCenter Server
[2011-05-04 11:25:13.222 04608 verbose 'ThreadPool'] TaskMax=90, IoMin=1, IoMax=
181
[2011-05-04 11:25:13.227 04608 info 'App'] Log path: C:\ProgramData\VMware\VMwar
e VirtualCenter\Logs
[2011-05-04 11:25:13.229 04608 info 'App'] Initializing SSL
[2011-05-04 11:25:13.229 04608 info 'Libs'] Using system libcrypto, version 9080
EF
[2011-05-04 11:25:14.271 04608 info 'App'] Vmacore::InitSSL: doVersionCheck = tr
ue, handshakeTimeoutUs = 120000000
[2011-05-04 11:25:14.273 04608 warning 'Libs'] W32Util_RegisterEventLog: Could n
ot open registry key. Error 0.
[2011-05-04 11:25:14.274 04608 warning 'Libs'] Failed to initialize event source
for VMware VirtualCenter Server.
[2011-05-04 11:25:21.361 04608 info 'App'] CoreDump: Writing minidump
[2011-05-04 11:25:22.274 04608 panic 'App']

Panic: Failed to do early initialization.
Backtrace:
backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFil
eWriter
backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQui
ckBacktrace
backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
backtrace[06] rip 000000013fc95793 (no symbol)
backtrace[07] rip 000000013f2abc58 (no symbol)
backtrace[08] rip 000000013f2ac0c4 (no symbol)
backtrace[09] rip 000000013f2aadf2 (no symbol)
backtrace[10] rip 0000000077aff56d BaseThreadInitThunk
backtrace[11] rip 0000000077c32cc1 RtlUserThreadStart

[2011-05-04 11:25:22.345 04608 panic 'App'] Failed to do early initialization.

------ In-memory logs end --------

D:\Program Files\VMware\Infrastructure\VirtualCenter Server>

Thanks in advance,

Steve

bgoering

Typically this is database related. Check database connectivity. Is the database local or remote? SQL or Oracle? Is it running?

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

any event log issues on the server in system or application logs?

how long have you been running vcenter, not using sql express, and the db is at the 4gb limit?

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Have a look at this kb article http://kb.vmware.com/kb/1008436 for some trouble shooting steps from the VMware knowledge base.

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Are you ODBC DSN settings correct and okay?

steve_fernandes

ASKER

Its a sql express installation on the box that vcenter itself is installed on.

ASKER CERTIFIED SOLUTION

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

bgoering

Try running

vpxd -p

To check/reset the db password. Is the Sqlservr service running?

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

whilst we are investigating this issue, you can still manage individual hosts, by using the hostname or ip address of the ESX server in vSphere Client GUI. (just in case you didn't know that!)

steve_fernandes

ASKER

I can manage individual host by using IP or Host name.
The dbs are all under 4 GB.

Sqlserver service is running.

The only thing I can think of that changed is that we used Backup Exec to start backup up the dbs for VCenter.

steve_fernandes

ASKER

Here are the results of the vpxd.exe -p

Its still erroring out:

D:\Program Files\VMware\Infrastructure\VirtualCenter Server>vpxd.exe -p
[2011-05-04 13:30:44.105 05692 info 'App'] Current working directory: D:\Program
Files\VMware\Infrastructure\VirtualCenter Server
[2011-05-04 13:30:44.111 05692 info 'App'] Log path: C:\ProgramData\VMware\VMwar
e VirtualCenter\Logs
[2011-05-04 13:30:44.112 05692 info 'App'] Initializing SSL
[2011-05-04 13:30:44.112 05692 info 'Libs'] Using system libcrypto, version 9080
EF
[2011-05-04 13:30:45.164 05692 info 'App'] Vmacore::InitSSL: doVersionCheck = tr
ue, handshakeTimeoutUs = 120000000
[2011-05-04 13:30:45.166 05692 warning 'Libs'] W32Util_RegisterEventLog: Could n
ot open registry key. Error 0.
[2011-05-04 13:30:45.167 05692 warning 'Libs'] Failed to initialize event source
for VMware VirtualCenter Server.
[2011-05-04 13:30:58.151 05692 info 'App'] CoreDump: Writing minidump
[2011-05-04 13:30:58.415 05692 panic 'App']

Panic: Failed to do early initialization.
Backtrace:
backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFil
eWriter
backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQui
ckBacktrace
backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
backtrace[06] rip 000000013ff65793 (no symbol)
backtrace[07] rip 000000013f57bc58 (no symbol)
backtrace[08] rip 000000013f57c0c4 (no symbol)
backtrace[09] rip 000000013f57adf2 (no symbol)
backtrace[10] rip 0000000077aff56d BaseThreadInitThunk
backtrace[11] rip 0000000077c32cc1 RtlUserThreadStart

[2011-05-04 13:30:58.486 05692 panic 'App'] Failed to do early initialization.

------ In-memory logs start --------
[2011-05-04 13:30:44.105 05692 info 'App'] Current working directory: D:\Program
Files\VMware\Infrastructure\VirtualCenter Server
[2011-05-04 13:30:44.105 05692 verbose 'ThreadPool'] TaskMax=90, IoMin=1, IoMax=
181
[2011-05-04 13:30:44.111 05692 info 'App'] Log path: C:\ProgramData\VMware\VMwar
e VirtualCenter\Logs
[2011-05-04 13:30:44.112 05692 info 'App'] Initializing SSL
[2011-05-04 13:30:44.112 05692 info 'Libs'] Using system libcrypto, version 9080
EF
[2011-05-04 13:30:45.164 05692 info 'App'] Vmacore::InitSSL: doVersionCheck = tr
ue, handshakeTimeoutUs = 120000000
[2011-05-04 13:30:45.166 05692 warning 'Libs'] W32Util_RegisterEventLog: Could n
ot open registry key. Error 0.
[2011-05-04 13:30:45.167 05692 warning 'Libs'] Failed to initialize event source
for VMware VirtualCenter Server.
[2011-05-04 13:30:58.151 05692 info 'App'] CoreDump: Writing minidump
[2011-05-04 13:30:58.415 05692 panic 'App']

Panic: Failed to do early initialization.
Backtrace:
backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFil
eWriter
backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQui
ckBacktrace
backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit
backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit
backtrace[05] rip 00000001800abfb4 Vmacore::Vpanic
backtrace[06] rip 000000013ff65793 (no symbol)
backtrace[07] rip 000000013f57bc58 (no symbol)
backtrace[08] rip 000000013f57c0c4 (no symbol)
backtrace[09] rip 000000013f57adf2 (no symbol)
backtrace[10] rip 0000000077aff56d BaseThreadInitThunk
backtrace[11] rip 0000000077c32cc1 RtlUserThreadStart

[2011-05-04 13:30:58.486 05692 panic 'App'] Failed to do early initialization.

------ In-memory logs end --------

D:\Program Files\VMware\Infrastructure\VirtualCenter Server>

Danny McDaniel

If you have any web services installed on the VC, that can cause issues, too. if you do 'netstat -anb' from the command prompt, look for any application listening on port 80 or 443

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Just check your DSN settings, does ODBC DSN connect successfully, and you just quickly run through the configuration again, and at the end select Test.

See if it connects successfully to the SQL server and db.

steve_fernandes

ASKER

No web services are installed on the VB. I ran netstat and did not find any for port 80 or 443.

I checked my DSN conenction settings and was able to connect perfectly to the local db server.

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

did the Backup Exec manage to backup them up, did you use an agent, make any changes to the db, ownership etc?

steve_fernandes

ASKER

we used the sql agent. I am left w/ a last option of doing a restore of the db and the logs for vcenter.

We also have veeam dbs installed on this server, so i probably should role these back as well?

No changes were made to the db or the ownerships as far as i can tell.

Danny McDaniel

you could just roll through an un-install/re-install of vCenter without overwriting the database. I don't see any ODBC messages in the logs like you'd normally see if there were problems with the database or even that it started connecting to it. That could be verified by enabling db tracing Enabling trivia logging in VMware vCenter and VirtualCenter (http://kb.vmware.com/kb/1001584)

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

You could always create a new database along side the old database, and create a new ODBC Entry, or use the old ODBC entry and point to the new db, and see if vCenter starts on a new db.

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

that would give you confirmation, that the db is broken, and not something else.

steve_fernandes

ASKER

Hey Hanccocka,

Whilst the DB had not reached the hard 4GB limit, the transaction log had reached the limit of 8GB. I changed the recovery model to simple and ran a shrink and all is well.

Thanks for pointing me in that direction,

Steve

Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

a couple of things either create a watch on the size of db or logs, or you can reduce the history statistics your are keeping in the vcenter settings. to try and avoid this, but all us admins have been in your shoes, because we will remember today, but when it stops its the first thing we ignore......a long time ago after a similar service outage, at a client site after discovering this issue for a client, the it department covered the VMware admin car in post it notes with the letters 4GB on it.

i kept one for the occassion, still on my monitor as a reminder!