VMWAre esxi 5.5 cluster error

You didn't attach the screenshot but based on the description I'm guessing that the agents are running into issues and the processes are hanging or dying and so they aren't communicating with vCenter (orphaned).  

You can try to restart the management agents. You can do this without having to shut down the VMs.

This KB lists the different ways you can restart the agents.

kb.vmware.com/kb/1003490

Check your SQL database, it's not full ?

SQL Express Database sizes are limited to 4GB and 10GB depending upon version.

It's only the Management Server which is "dying" so don't panic!

This will not affect the running of the VMs, which is important.

Also try restarting the vCenter Server SERVICE.... on Windows, if you are using Windows OS.

either via RDP or Console (via ESXi direct)

I looked at the article and will attempt the restart of the management agents. These are in a data center so I will have to see if I can log on and turn on SSH. I am attampting to upload the screenshot again to verify the issue.
Untitled.jpg

The lack of CPU and Memory detailed in the summary, indicates an Agent issue.

Just check SQL database.

I'm seeing three hosts not communicating with vCenter. If restarting the agents doesn't help, I'd begin to investigate the storage next or maybe even there's a network communication issue that's causing this. Keep us posted.

I kind of doubt there is an issue with the management agents on all hosts. Do you have the root password or an account that can log directly into each ESXi host from vSphere? At least this way you can verify if there is an issue with the host. The easiest way to restart management agents is to SSH into the host, if SSH is enabled, and run "services.sh restart".

Open up the vSphere Client and log into 192.168.5.215 or another host directly instead of going through vCenter. Based on your first comment it doesn't sound like you are actually logging directly into the host, but rather just selecting it from within vCenter. There is a big difference.

Do you have vCenter installed on a Windows Server or are you using a vCenter Server Appliance? Have you tried restarting vCenter? This has no affect on the hosts or VMs running.

Also, can you connect to any of the servers that are running as VMs? Such as RDP to a Windows Server or SSH to a Linux server? If all the servers are running and fine, then there is likely no storage issue either.

All the VM's are running and I am able to connect to them using remote desktop. I can log into the hosts themselves and there are no errors that show on either. It only gives the error when logging in through vcenter,

Did you restart agents?

restart vCenter Server Service?

Check SQL DB size ?

It would also be worth checking that the Hosts can ping vCenter Server by IP Address and Hostname? (and vica versa)

No communication issues, or firewalls that have gone up...

Which is what I expected. You need to troubleshoot vCenter and connectivity from vCenter to the ESXi hosts. Such as making sure you don't have ports blocked that is causing a communication issue from vCenter to the ESXi hosts.

Again, is vCenter running on Windows or is it an appliance? Have you restarted the vCenter server? Are all services running? If running on Windows, check the event logs. You can check the database too, but generally speaking of the DB is full, the vCenter service will stop running and you won't even be able to log into vCenter.

Totally agree, looks like you've narrowed it down to the vCenter side, KapsZ28 and Andy have good ideas on what to check. I'll sit by as moral support now. :)

I did restart the agents using /etc/init.d/hostd restart and /etc/init.d/vpxa restart on both hosts with no affect. I have rebooted the vcenter server several times with no affect. The SQL database size looks to be about 11 gigs. And vcenter is running as VM on the cluster.

Are you using SQL Express or SQL Full ?

e.g. did you use the SQL which is included with the installation of vCenter Server 5.5 for Windows.

SQL Express 2012 is limited to a maximum of 10GB

see here

http://sqlmag.com/sql-server-2012/sql-server-2012-express-editions

You may need to shrink your database, if this is SQL Express. e.g. the free version, that vCenter Server 5.5 installation comes with!)

I used the SQL that was included with the vcenter server for windows, it is 2008 R2

I am not a SQL anything hehe. But if you let me know what to run I am more then happy to try.

You should at least look at the database file before trying to shrink it. What is the current size of the .MDF?

get me a quick screen grab of the data folder for SQL, and version....and just before I head off to my bed, here in the UK (00:38), you may need to download and install SQL Management studio, to look and manage your dbs.

be back in 5 hrs!

Here is a screenshot of the database folder, but it did not install SQL manager so not sure how to run those scripts
Untitled.jpg

Yes, database too large.

SQL Management studio

you will need this download

https://www.microsoft.com/en-gb/download/details.aspx?id=29062

You will need to install it on a workstation, or vCenter Server, to open the SQL Instance, to Execute the Code, I've uploaded here...

After running script to shrink the database, and restarting the agents, vcenter was working 100%. Thank you everyone.

Yay!