Solved

VMware vCenter service randomly stops with error: "The VMware VirtualCenter Server service terminated with service-specific error The system cannot find the file specified.. "

Posted on 2013-11-28
10
4,706 Views
Last Modified: 2013-12-09
VMware vCenter service randomly stops with error:
 "The VMware VirtualCenter Server service terminated with service-specific error The system cannot find the file specified.. "

From vpxd.log on vCenter:
        2013-11-24T16:46:24.229+01:00 [10540 error 'Default'] SSLStreamImpl::DoServerHandshake (0000000017201230) SSL_accept failed. Dumping SSL error queue:
        2013-11-24T16:46:24.229+01:00 [10540 error 'Default'] [0] error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
        2013-11-24T16:46:24.229+01:00 [10540 warning 'ProxySvc'] SSL Handshake failed for stream TCPStreamWin32(socket=TCP(fd=916) local=<vcenter IP here>:443,  peer=<vcenter IP here>:57963), error: class Vmacore::Ssl::SSLException(SSL Exception: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown)
        2013-11-24T16:46:24.697+01:00 [10912 info 'commonvpxLro' opID=995a566] [VpxLRO] -- BEGIN task-internal-53548 --  -- vmodl.query.PropertyCollector.retrieveContents -- 7635ab46-1871-a626-0fa0-6de4d11dcc06(522bf4cd-7972-c569-3881-1f8775ac9a65)
        2013-11-24T16:46:24.697+01:00 [10912 info 'commonvpxLro' opID=995a566] [VpxLRO] -- FINISH task-internal-53548 --  -- vmodl.query.PropertyCollector.retrieveContents --
        2013-11-24T16:46:24.728+01:00 [10676 info 'commonvpxLro' opID=f975d9a9] [VpxLRO] -- BEGIN task-internal-53549 --  -- vmodl.query.PropertyCollector.retrieveContents -- d5897125-0519-d465-addc-04bceacc31ed(5293f85d-c2dc-e257-d6fd-f5256f629d8d)
        2013-11-24T16:46:24.728+01:00 [10676 info 'commonvpxLro' opID=f975d9a9] [VpxLRO] -- FINISH task-internal-53549 --  -- vmodl.query.PropertyCollector.retrieveContents --
        2013-11-24T16:46:29.970+01:00 [10360 info 'Default' opID=HB-host-21784@121322-e52759ae] Forcing shutdown of VMware VirtualCenter now
       
Verified all steps here:
        http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003926
       
Verified this:
        http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2056296
       
Ensured this:
        http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020210

Still the problem persists. When the service is started manually, it runs for a time period of less then 24 hours up to two days. Typically less then 24 hours. It fails in the middle of the night, or at the end of the work day judging by the times seen so far.
0
Comment
Question by:itnifl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
10 Comments
 
LVL 5

Expert Comment

by:Steve M
ID: 39683928
A couple questions...
SQL server or SQL Express?  (on this server or another server?)
Do you have Antivirus running on this server and/or the SQL server?
0
 
LVL 2

Author Comment

by:itnifl
ID: 39684701
Using SQL 2008 R2 on a separate server.
Have Deep Security 9 SP1 Appliance protection on the SQL server and vCenter.
0
 
LVL 5

Assisted Solution

by:Steve M
Steve M earned 500 total points
ID: 39685321
Have you tried excluding the AV scans from scanning the vcenter files?  I assume the SQL databases are already excluded from your AV scans, (but if not they should be).
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 
LVL 2

Author Comment

by:itnifl
ID: 39695207
OK - I have done exclusions as described in:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2010667

Both:
C:\Program Files\VMware\Infrastructure\Inventory Service\
C:\ProgramData\VMware\Infrastructure\Inventory Service\

Will have to see if that does the trick.
0
 
LVL 5

Expert Comment

by:Steve M
ID: 39695671
Great, hope it works for you!

(BTW, did you exclude from backups as well if applicable?)

Good Luck!
Steve
0
 
LVL 2

Author Comment

by:itnifl
ID: 39700453
I have made an request to exclude from backups as well.
In addition I found several of the following in the application log:

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (42000) - [Microsoft][SQL Server Native Client 10.0][SQL Server]The transaction log for database 'VIM_VCDB' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases" is returned when executing SQL statement "IF EXISTS(SELECT 1 FROM VPX_PROPERTY_BULLETIN WHERE OBJECT_TYPE=? AND OBJECT_MOID=? AND PROPERTY_PATH=?)  UPDATE VPX_PROPERTY_BULLETIN WITH (ROWLOCK) set GEN_NUMBER = ?, MOD_TIME = ?, OPERATION_TYPE = ? WHERE OBJECT_TYPE=? AND OBJECT_MOID = ? AND PROPERTY_PATH = ? else  INSERT INTO VPX_PROPERTY_BULLETIN WITH (ROWLOCK) (GEN_NUMBER, OBJECT_TYPE, OBJECT_MOID, PROPERTY_PATH, OPERATION_TYPE, MOD_TIME) values (?, ?, ?, ?, ?, ?)"

the message resource is present but the message is not found in the string/message table

So I truncated the log file for the VIM_VCDB database. It was also set with a growth limit to 500MB. I set it to unlimited. If the disk runs nearly full, I will be alerted by our surveillance system and then I can trunk the log again. Or maybe better to schedule a periodic truncation withing SQL server? Either way, will have to give things some time and see how this goes.
0
 
LVL 5

Accepted Solution

by:
Steve M earned 500 total points
ID: 39700908
Have a read through this post - it may help in your situation with the full logs

https://communities.vmware.com/thread/345305

/Steve
0
 
LVL 2

Author Comment

by:itnifl
ID: 39701002
Thanks. In my case VPX_EVENT has a row count of 192125 and takes 45.58 MB of space.
VPX_EVENT_ARGS has a row count of 833547 and takes 245.539 MB of space. I guess more truncation is not needed?
0
 
LVL 2

Author Closing Comment

by:itnifl
ID: 39705534
Not sure if it was exclusion of av-scans or truncations of logs that did it, but both are probably good suggestions.
0

Featured Post

Ransomware: The New Cyber Threat & How to Stop It

This infographic explains ransomware, type of malware that blocks access to your files or your systems and holds them hostage until a ransom is paid. It also examines the different types of ransomware and explains what you can do to thwart this sinister online threat.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
VMWare Calculate number of processors 10 101
Windows Server get hanged and Reset of network adpapter required 3 66
vmware, windows server 2012 10 64
Datacenter object VMWare 7 63
In this article, I show you step by step with screenshots to assist you - HOW TO: Deploy and Install the VMware vCenter Server Appliance 6.5 (VCSA 6.5), with some helpful tips along the way.
In this article we will learn how to backup a VMware farm using Nakivo Backup & Replication. In this tutorial we will install the software on a Windows 2012 R2 Server.
Teach the user how to configure vSphere Replication and how to protect and recover VMs Open vSphere Web Client: Verify vsphere Replication is enabled: Enable vSphere Replication for a virtual machine: Verify replicated VM is created: Recover replica…
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question