Link to home
Start Free TrialLog in
Avatar of montypy
montypy

asked on

How to trace the cause of Out Of Memory issues

Is there a way to trace what is causing out of memory kernel panics? Server keeps rebooting
Avatar of montypy
montypy

ASKER

anyone?
If the log files and panic messages don't give you any clues, you could try running a tool like top or vmstat to see if you can spot a runaway process as the culprit. Once you can identify a particular program as the cause, it should be possible to drill down and find out why it's misbehaving. Depending on what's going wrong, allocating a much larger pagefile may slow the runaway and delay the crash so as to give you more of a window to catch the leaker, if it is a leak.

Knowing that your server's configuration is adequate for the workload is sometimes hard. If you can run the server image in a virtual machine, more tools can be brought to bear. If it originated as a pre-packaged VM, it's possible the default configuration is just too stingy for it to do its job: e.g. the Turnkey Linux packages, at least the ones I've tried, start off using just a quarter-GB of RAM. This may have to be increased if the workload they're doing turns out to require more RAM.
Avatar of arnold
Panic usually means there are faulty memory modules.
I.e. There is a point where the consumed memory hits the faulty modules triggering the kernel panic event.

Is this a new or old server? Deals with whether the voltage across the memory modules is marginal below required but within the 5, 10% allotment until at one point it dips below that threshold destabilizing the data in memory responses triggering a kernel panic.

Please post the event from /var/log/messages dealing with the panic event as well as 10-20 lines preceding it.

Please include details what this server does, functions, etc.


The simplest thing, during bootup, run memtest, often installs include this test. See if the hardware includes a test as well.
This raises a question. If you are getting a panic that's specifically labeled "out of memory", then I think it's unlikely to be caused by faulty memory modules. Possible, but unlikely.

A few questions that need to be answered: 32-bit or 64-bit kernel? Physical or VM? How much physical RAM installed in the server (or virtual RAM allocated to the VM). What's the exact text of the panic message? Can you show the tail end of /var/log/messages from the point of the panic backwards?

Since the server is crashing more-or-less continually, it doesn't hurt to run memory diagnostics for a few rounds. I'm just a bit dubious that you'll find the source of this problem in that direction.
ASKER CERTIFIED SOLUTION
Avatar of montypy
montypy

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
So, when setting this server up, was some step skipped to cause memcached to be started up and initialized? There's sort of a loose coupling that I supposed someone might miss.

Glad you were able to resolve it. You can choose whether you want to ask for the question to be deleted or to accept your ultimate resolution as a 0-point solution.
Avatar of montypy

ASKER

I guess it was skipped and yes very glad we caught it.
Avatar of montypy

ASKER

Found the answer.