VM crashing intermittently

Thanks Andrew.  That was going to be the end result but is there a log somewhere in ESXi that will tell me exactly why the server is powering off?

The OS is Server 2012 Std.  Supposedly the guy before me ran diags.  Note, server2 stays up fine until server1 tanks the whole thing.
Thanks, I'll check out the logs.

So when I opened up the server, I discovered they never enabled the second processor on the board.  I switched the jumper and moved the RAM around but my question is now, will the VMs use that second processor automatically when it's available or do I have to configure something?  I'd like each vm to use a separate processor, right?

CPU/MMU virtualization is set to automatic.  Also, the Swapfile location is set to default.  What would provide the best performance for that?

So when I opened up the server, I discovered they never enabled the second processor on the board.  I switched the jumper and moved the RAM around but my question is now, will the VMs use that second processor automatically when it's available or do I have to configure something?  I'd like each vm to use a separate processor, right?

This would cause hardware issues, ESXi will use the memory and processor automatically. Also remember that VMware vSphere is licensed per socket, if you do not have the correct licenses, the license will be violated.

You may want to check what you are licensed for, if this is a paid for license.

Also make sure the memory is balanced between processors, and correct for the sockets.

CPU/MMU virtualization is set to automatic.  Also, the Swapfile location is set to default.  What would provide the best performance for that?

These are default values and normal.

I checked the vSphere and it says VMware vSphere 5 Essentials Licensed for 2 physical CPUs (unlimited cores per CPU).  So I don't need to change anything to utilize both of them?  It doesn't seem like any of this should have worked using one CPU if,now with two cpus, Server 1 was pegged at 95%.  It has calmed down a bit since posting and server 1 CPU utilization is steady around 45%, bouncing up around 80%.  Wondering if maybe CPU maxing out is crashing the VM but doesn't make sense that it causes the ESXi host to crap out and not respond as well.

I checked the vSphere and it says VMware vSphere 5 Essentials Licensed for 2 physical CPUs (unlimited cores per CPU).  So I don't need to change anything to utilize both of them?

No, you are fine.

It doesn't seem like any of this should have worked using one CPU if,now with two cpus, Server 1 was pegged at 95%.  It has calmed down a bit since posting and server 1 CPU utilization is steady around 45%, bouncing up around 80%.  Wondering if maybe CPU maxing out is crashing the VM but doesn't make sense that it causes the ESXi host to crap out and not respond as well.

If a server has not been configured correctly, and a CPU exists in the socket, and it's been disabled, this is not a normal installation for a Hypervisor, so anything could happen.

I think now you have changed the configuration, you need to test, test and test again, and observe, if the issues are repeated.

There are situations where a ESXi Hypervisor, CPU and Memory can max out, and the host does come unresponsive.

I do not think you have given any information as to

1. HOST Specification, e.g. memory and cpus.

2. VM specification.

e.g. if you have allocated silly CPU and Memory figures to VMs, this can have the effect you describe.

Well we upgraded the firware on drives 2-4 that have the main datastore on them and turned on that second CPU and were good for a day but it Server1 crapped out twice yesterday, made the Host stop responding so we couldn't connect remotely, and an hour or so later, Server 2 bombs out and we have to do a power cycle.
Server1:  
RAM:  10240MB
CPUs4
Video Card: Video Card
VMCI device:  Restricted
SCSI Controller 0: LSI Logic Parallel
Hard Disk 1 : Virtual Disk
CD/DVD drive 1:  Client Device
Network Adapter 1:  Data Network

Server 2:  
RAM: 8192MB
CPUs: 4
Video Card: Video Card
VMCI device:  Restricted
SCSI Controller 0: LSI Logic Parallel
Hard Disk 1 : Virtual Disk
CD/DVD drive 1:  Client Device
Network Adapter 2:  Data Network

Sorry,
Server1:  
SCSI Controller 0:  LSI Logic SAS

and the host CPUs and Memory..

So total memory assign to VMs is 18GB

and 8 vCPUs (which could be alot!)

What are the servers function ?

and could you explain, what happnes in more detail when the server "boms out" - that's not really a technical term!

see here

vSMP (virtual SMP) can affect virtual machine performance, when adding too many vCPUs to virtual machines that cannot use the vCPUs effectly, e.g. Servers than can use vSMP correctly :- SQL Server, Exchange Server.

This is true, many VMware Administrators, think adding lots of processors, will increase performance - wrong! (and because they can, they just go silly!). Sometimes there is confusion between cores and processors. But what we are adding is additional processors in the virtual machine.

So 4 vCPU, to the VM is a 4 Way SMP (Quad Processor Server), if you have Enterprise Plus license you can add 8, (and only if you have the correct OS License will the OS recognise them all).

If applications, can take advantage e.g. Exchange, SQL, adding additional processors, can/may increase performance.

So usual rule of thumb is try 1 vCPU, then try 2 vCPU, knock back to 1 vCPU if performance is affected. and only use vSMP if the VM can take advantage.

Example, VM with 4 vCPUs allocated!

My simple laymans explaination of the "scheduler!"

As you have assigned 4 vCPUs, to this VM, the VMware scheulder, has to wait until 4 cores are free and available, to do this, it has to pause the first cores, until the 4th is available, during this timeframe, the paused cores are not available for processes, this is my simplistic view, but bottom line is adding more vCPUs to a VM, may not give you the performance benefits you think, unless the VM, it's applications are optimised for additional vCPUs.

See here
http://www.vmware.com/resources/techresources/10131

see here
http://www.gabesvirtualworld.com/how-too-many-vcpus-can-negatively-affect-your-performance/

http://www.zdnet.com/virtual-cpus-the-overprovisioning-penalty-of-vcpu-to-pcpu-ratios-4010025185/

also there is a document here about the CPU scheduler

www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf

https://blogs.vmware.com/vsphere/2013/10/does-corespersocket-affect-performance.html

The VMware.log might as well be in Chinese.  
02T22:33:52.052Z| vmx| I120: OvhdMem: memsize 10240 MB VMK fixed 74 pages var 0 pages cbrcOverhead 0 pages total 5199 pages
2015-04-02T22:33:52.052Z| vmx| I120: VMMEM: Maximum Reservation: 165MB (MainMem=10240MB SVGA=4MB) VMK=20MB
2015-04-02T22:33:52.087Z| vmx| I120: VMXVmdb_SetToolsVersionStatus: status value set to 'ok', 'current', install possible
2015-04-02T22:33:52.090Z| vmx| I120: Destroying virtual dev for scsi0:0 vscsi=8192
2015-04-02T22:33:52.090Z| vmx| I120: VMMon_VSCSIStopVports: No such target on adapter
2015-04-02T22:33:52.092Z| vmx| I120: MKS PowerOff
2015-04-02T22:33:52.102Z| vmx| I120: scsi0:0: numIOs = 0 numMergedIOs = 0 numSplitIOs = 0 ( 0.0%)
2015-04-02T22:33:52.102Z| vmx| I120: Closing disk scsi0:0
2015-04-02T22:33:52.129Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/50e5f795-df7cb1ce-fc16-ac162db20a54/LSServer1/LSServer1-000001-delta.vmdk" : closed.
2015-04-02T22:33:52.146Z| vmx| I120: DISKLIB-VMFS  : "/vmfs/volumes/50e5f795-df7cb1ce-fc16-ac162db20a54/LSServer1/LSServer1-flat.vmdk" : closed.
2015-04-02T22:33:52.278Z| vmx| I120: WORKER: asyncOps=2 maxActiveOps=1 maxPending=0 maxCompleted=1
2015-04-02T22:33:53.392Z| vmx| I120: Vix: [5763 mainDispatch.c:3867]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1 additionalError=0
2015-04-02T22:33:53.392Z| vmx| I120: Vix: [5763 mainDispatch.c:3886]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.

2015-04-02T22:33:53.533Z| vmx| I120: Vix: [5763 mainDispatch.c:3867]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=1 additionalError=0
2015-04-02T22:33:53.533Z| vmx| I120: Vix: [5763 mainDispatch.c:3886]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
2015-04-02T22:33:53.534Z| vmx| I120: Transitioned vmx/execState/val to poweredOff
2015-04-02T22:33:53.534Z| vmx| I120: VMX idle exit
2015-04-02T22:33:53.534Z| vmx| I120: VMIOP: Exit
2015-04-02T22:33:53.808Z| vmx| I120: Vix: [5763 mainDispatch.c:861]: VMAutomation_LateShutdown()
2015-04-02T22:33:53.808Z| vmx| I120: Vix: [5763 mainDispatch.c:811]: VMAutomationCloseListenerSocket. Closing listener socket.
2015-04-02T22:33:53.811Z| vmx| I120: Flushing VMX VMDB connections
2015-04-02T22:33:53.811Z| vmx| I120: VmdbDbRemoveCnx: Removing Cnx from Db for '/db/connection/#1/'
2015-04-02T22:33:53.811Z| vmx| I120: VmdbCnxDisconnect: Disconnect: closed pipe for pub cnx '/db/connection/#1/' (0)
2015-04-02T22:33:53.817Z| vmx| I120: VMX exit (0).
2015-04-02T22:33:53.817Z| vmx| I120: AIOMGR-S : stat o=2 r=6 w=0 i=0 br=98304 bw=0
2015-04-02T22:33:53.817Z| vmx| I120: OBJLIB-LIB : ObjLib cleanup done.
2015-04-02T22:33:53.817Z| vmx| W110: VMX has left the building: 0.


Is that saying April 2?  Because that is where the log stops.

"What are the servers function?"

Server1 is a domain controller and hosts an SQL database for a program called the users call Seradex but the computer calls it ActiveERP.  It is has a db called Seradex.  

When it "bombs out" we can't see anything.  It just goes offline and we can't get to it remotely.  That is why I was looking for logs.

"Is that saying April 2?"
Duh, yeah, after I expanded the datastore browser, it says the VMware.log was last modified on 4.2.2015 on server1 (MainDataStore).

It's not good practice to have anything share a Domain Controller.

So what does Server2 do ?

What is the HOST CPU and MEMORY ?

HOST specifications.

you should have other vmware*.logs ?

do pings stop ?

does RDP stop ?

vSphere Client connected to ESXi server?

Console Output ?

I've also noticed your VMs is running on a snapshot, which is not good....also..see my EE Article

HOW TO: VMware Snapshots :- Be Patient

do you have free space on the datastore ?

if the snapshot fills the datastore, the VM will stop. also performance of the VM on a snapshot will be poor.

"It's not good practice to have anything share a Domain Controller."

That's what I've been saying.  Server1 and Server2 are DCs.  Server2 is the DHCP and DNS server as well and is a license server for Solidworks.  What is the point of having your domain controller and its backup on the same vm host, you ask?  Yeah, me too.

"What is the HOST CPU and MEMORY ?"
8 CPUs x 1.795 GHz
Intel Xeon E5
Memory:  24541.20 MB
MainDataStore:  1.36TB  Free:  81.88GB
DataStore2:  1.36TB  Free  1.21TB

Wait, I just noticed something after typing that.
Server2:  
Provisioned Space:  158.79GB
Used Space:  158.79GB
Host CPU MHz:  2064
Mem:  8083

Server1:
Provisioned Space 2.93TB
Used:  1.28TB
Host Cpu:  2850
Mem:  10311

Yet in resource allocation it says:  
Server2   Hard Disk 1  DataStore2
Server1   Hard Disk 1  MainDataStore

Disks.docx

Disks.docx

Pings don't stop and Server2 stays up for about an hour but RDP stops as does vSphere and console output.

Snapshot?

Snapshots don't often appear in the snapshot manager.

check the disks.

Alright, well I'm waiting for somebody to buy the new drives, I guess.  In the meantime, server1 hasn't crashed since I messed around with it.  Not sure why.  I'll close this issue since a rebuild is in order.  Oh one more thing, what did you see that made you think it was running on a snapshot?  I've been all over the place and haven't looked in the vm folder but was curious as to what you saw.

You are right though.  Server1.vmdk hasn't been modified since 4/1/15 and server1-0000001.vmdk exists.  What steps would you recommend to properly merge the two and get a good backup before I wipe it and start over?

First check, you have enough disk space for the Snapshot Merge.

1. Create a new Snapshot (Take Snapshot)
2. Wait 60 seconds
3. Select DELETE ALL and Be Very Patient.

How large is the snapshot ? you can check via datastore browser ?

Sorry for the delay.  Server1.vmdk is 1.3TB and the Server1-000001.vmdk is 56.5GB.

Okay, go ahead with my last post.

This thing is also showing me an alarm for Power Supply 3 and lost redundancy.  This server doesn't even have 2 power supplies let alone 3.  How do I disable this alarm?

In the datastore details, it shows MainDataStore has a capacity of 1.36TB and 78.53GB free.  Will it be ok?

Ok to do the snapshot, that is?

Thanks Andrew.  HP is sending us a new motherboard and we bought 8 new 1TB hard drives.  Will be utilizing the solution outlined above this Friday night.  Fingers crossed.