How to clear out an ESXi log file that is full?

Hello:

I received an alert that a system log file is close to being full on our Dell PowerEdge T710 server.

screen1
screen2
I cleared out the event logs from the Vsphere client.  I cleared out the event logs from iDRAC; but, the alerts is still showing up.  I have re-set the sensors and the alert still comes back.  What can I do to clear up the event logs for this ESXi server?
LVL 1
PkafkasNetwork EngineerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Zephyr ICTCloud ArchitectCommented:
Yeah, we had this one to in version 5.x, we couldn't get rid of it, sometimes it would stay gone for a few days other times it came back each time we did the necessary steps to clear the logs following this article from VMware

I don't know which version you have, but you could try that article to clear it. Also make sure to click the "reset sensors" at the top right of you vsphere client in system event log.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
This is a common issue with servers using ESXi.

If you cannot reset it through ESXi (you are using the OEM Dell version of ESXi).

at next host server reboot, go into the BIOS and Reset there.
0
PkafkasNetwork EngineerAuthor Commented:
Hello Spravtek:

I have already attempted the instructions  that are in the web address, that you suggested.  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033725

We do have VMware 5.5 update 2 running on this ESXi host.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

PkafkasNetwork EngineerAuthor Commented:
Hello Andrew Hancock,

We have local storage on this Server and rebooting the server means powering off all of the VM's and then powering them back on.  It will have to be a planned maintenance window to go into the bios and to reset the event log in there.

Before I power off the ESXi host, I must know

1.  Where to re-set the event log in the BIOS.  How to make sure that the event log is where Host IPMI is what it is reffering to.
0
Zephyr ICTCloud ArchitectCommented:
Hi Pkafkas,

Yes, it was a long shot, the only thing left is rebooting like Andrew mentioned and clear the log that way.
Problem is it can come back, like I said, dealt with the same thing, also version 5.5, even when clearing the log via reboot it came back. It was HP hardware though, so not entirely the same.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We have local storage on this Server and rebooting the server means powering off all of the VM's and then powering them back on.  It will have to be a planned maintenance window to go into the bios and to reset the event log in there.

Yes, that's correct, unless you are running a large cluster with vMotion, and then it does not matter!

Are you using the OEM DELL version ?
0
PkafkasNetwork EngineerAuthor Commented:
I think I may try to connect to the Dell ESXi host via SSH and run a command.

there is a new localcli command line to clear the IPMI SEL logs:

    Run this command:

    "localcli hardware ipmi sel clear"

if I recall correctly, I will need to enable SSH on the host itself.  Will that put the host in maintenance mode?  Or cause the host to indicate a warning of some sort?
0
PkafkasNetwork EngineerAuthor Commented:
Hello Andrew Hancock:

I am not sure what you mean by your question "Are you using the OEM DELL version ?"  We are using Vmware 5.5 update 2.
0
Zephyr ICTCloud ArchitectCommented:
Yes, it's mentioned in the VMware KB, the command... "localcli hardware ipmi sel clear"

If SSH is not enabled you will need to enable it indeed, try the command, I seem to remember it worked, but it's been a while.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
I am not sure what you mean by your question "Are you using the OEM DELL version ?"  We are using Vmware 5.5 update 2.

Where did you download your version of ESXi from

1. VMware
2. Dell

if you downloaded from 1. VMware, I'm afraid you downloaded the wrong one, you should be using the OEM version of ESXi from DELL!!!!!!

because it includes the correct hooks to hardware to cope with things like this!

No need to put host in Maintenance Mode, enabling SSH will not cause VM issues, by it will cause a Warning! Notice on the server.

see my EE Articles

Part 5: HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 5.1 (ESXi 5.1)


HOW TO: Suppress Configuration Issues and Warnings Alert displayed in Summary status for ESXi 4.1 U2 or ESXi 5.0 after enabling SSH or ESXi Shell

(same procedure for 5.5! - articles were written before 5.5 was released!)

see also here
Dell's Customization of VMware ESXi and Its Advantages

You are not alone, it's very common for VMware Admins, to not use the OEM version!
0
PkafkasNetwork EngineerAuthor Commented:
I will ask the consultant  where he got the VMware 5.5 update 2 from.
0
PkafkasNetwork EngineerAuthor Commented:
Andrew Hancock, I can enable the SSH from the ESXi  host screen right correct?  

If I use your document of if I enable and then disable SSH fro the ESXi host screen, it will work the same way, id that right?  (http://www.experts-exchange.com/articles/11651/HOW-TO-Enable-SSH-Remote-Access-on-a-VMware-vSphere-Hypervisor-5-1-ESXi-5-1.html?printer=true )
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Andrew Hancock, I can enable the SSH from the ESXi  host screen right correct?  

If I use your document of if I enable and then disable SSH fro the ESXi host screen, it will work the same way, id that right?  (http://www.experts-exchange.com/articles/11651/HOW-TO-Enable-SSH-Remote-Access-on-a-VMware-vSphere-Hypervisor-5-1-ESXi-5-1.html?printer=true )

yes, you can do that!
0
PkafkasNetwork EngineerAuthor Commented:
I enabled SSH on the host and I ran the comand via putty.

screen11

But the event log went away and then it came back again.  I think the only other thing to try is to clear the log from the BIOS.  Unless i did nor execute the correct command.
0
Zephyr ICTCloud ArchitectCommented:
But the event log went away and then it came back again.  I think the only other thing to try is to clear the log from the BIOS.  Unless i did nor execute the correct command.

I was afraid of that, we had the same thing, yes rebooting will now be the only (possible) solution left, but it can still come back... There's no harm in it safe from being annoying. Some people disable the alert altogether... Not saying you should.
0
PkafkasNetwork EngineerAuthor Commented:
I know for HP ESXi hosts (Specifically ProLiant Gen 8) we can restart teh management services and that does not require the VM's to go off-line.  Is there anything like that, restarting the management services on the Dell T710, that can be done?

Otherwise, since we have local storage on this ESXi Host, we will need to power off all of the Virtual Servers and restart the ESXi host.  Then clear the logs from the BIOS.  Then restart the ESXi host again and then power on all of the VM's.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Restart Management Agents can be done on all hosts.....it does not affect VMs.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
PkafkasNetwork EngineerAuthor Commented:
So I now have 2 questions.

1.  How may I restart the management agents on a Dell PowerEdge T710 Server?
        a.  I can also ask the vendor that we have support from.
        b.  Even though they have not gotten back to me in a couple of days.


2.  How can I find the option to clear the logs int eh BIOS, of the Dell T710?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
1. either from the Console Menu - Restart  Network Management Agents.

or at the console or SSH (remote)

/sbin/services.sh restart

2. After you press F2 to Enter Setup, there should be an option to Clear Events.
0
PkafkasNetwork EngineerAuthor Commented:
Will restarting management services affect the VM's that are on the ESXi host?  Or will I have to plan a maintenance window just in case it brings down any Virtual servers?
0
Zephyr ICTCloud ArchitectCommented:
It may impact tasks that are running at the time you restart the management services and there are some other things to watch out for, check this KB for more info. But in principle you should be able to restart them without too much impact.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Restarting Network Management Agents for what purpose though ?
0
PkafkasNetwork EngineerAuthor Commented:
In the past, with an HP ESXi Server, a false alert, just like what I have now went away only after I restarted the management agents.
0
PkafkasNetwork EngineerAuthor Commented:
Thanks I will try that out and keep the thread going.  I hope powering off the ESXi host will be a last resort.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
If the logs are FULL, the logs are FULL, the Alerrt will come back!
0
PkafkasNetwork EngineerAuthor Commented:
My theory is that the logs are not full, iDrac has cleared them out.  The sensors just need to be reset and perhaps restarting the management agents could help that with the vsphere client.  we will see, it is easier to try then to power off the ESXi host.

Especially with local storage because I will have to power off all of the VM's before hand.  The pray that they come back up and work just as they were working before.

Restarting the management agents actually did clear out a false alert on an HP ESXI host.
0
Zephyr ICTCloud ArchitectCommented:
Yes, like I said earlier, I had this once on an environment using HP hardware, couldn't get rid of it even after clearing the logs, but for me the 'localcli' command did work, but I had to repeatedly do it again... In the end an upgrade of the VMware vsphere environment solved the issue.
0
PkafkasNetwork EngineerAuthor Commented:
It appears that restarting the management agents cleared out the false error that is shown above (question description).  It has been 10 hours and the false error has not come back.  I would like to wait at least 24 hours before I declare that it is fixed.
0
PkafkasNetwork EngineerAuthor Commented:
The alerts have not come back.  It appears that re-setting the logs from the Idrac and then resetting the management agents did the trick.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.