[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.
Question
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!
9.8

OOM killer strikes

Asked by medent in Linux Administration, Red Hat Linux

Tags: oom, killer

Hi,
I have a Centos linux 4.5 server loaded with 64G of Memory, and the Oom killer has kicked on a few times while there seems to be plenty of free mem , at least from what TOP shows.  Not sure what it is running low on and how I can adjust...?

Here is a snip from messages during an OOM condtion:
Feb 18 15:05:54 fox kernel: oom-killer: gfp_mask=0xd0
Feb 18 15:05:54 fox kernel: Mem-info:
Feb 18 15:05:54 fox kernel: DMA per-cpu:
Feb 18 15:05:54 fox kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: cpu 2 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 2 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: cpu 3 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 3 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: cpu 4 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 4 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: cpu 5 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 5 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: cpu 6 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 6 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: cpu 7 hot: low 2, high 6, batch 1
Feb 18 15:05:54 fox kernel: cpu 7 cold: low 0, high 2, batch 1
Feb 18 15:05:54 fox kernel: Normal per-cpu:
Feb 18 15:05:54 fox kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 1 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 1 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 2 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 2 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 3 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 3 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 4 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 4 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 5 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 5 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 6 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 6 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 7 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 7 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: HighMem per-cpu:
Feb 18 15:05:54 fox kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 1 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 1 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 2 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 2 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 3 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 3 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 4 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 4 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 5 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 5 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 6 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 6 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel: cpu 7 hot: low 32, high 96, batch 16
Feb 18 15:05:54 fox kernel: cpu 7 cold: low 0, high 32, batch 16
Feb 18 15:05:54 fox kernel:
Feb 18 15:05:54 fox kernel: Free pages:    15134404kB (15117568kB HighMem)
Feb 18 15:05:54 fox kernel: Active:6802704 inactive:5932559 dirty:1 writeback:0 unstable:0 free:3783601 slab:82688 mapped:1420230 pagetables:32180
Feb 18 15:05:54 fox kernel: DMA free:13124kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:20566 all_unreclaimable? yes
Feb 18 15:05:54 fox kernel: protections[]: 0 0 0
Feb 18 15:05:54 fox kernel: Normal free:3712kB min:3728kB low:7456kB high:11184kB active:792kB inactive:452kB present:901120kB pages_scanned:1719 all_unreclaimable? yes
Feb 18 15:05:54 fox kernel: protections[]: 0 0 0
Feb 18 15:05:54 fox kernel: HighMem free:15117568kB min:512kB low:1024kB high:1536kB active:27210024kB inactive:23729784kB present:68288512kB pages_scanned:0 all_unreclaimable? no
Feb 18 15:05:54 fox kernel: protections[]: 0 0 0
Feb 18 15:05:54 fox kernel: DMA: 1*4kB 2*8kB 5*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 3*4096kB = 13124kB
Feb 18 15:05:54 fox kernel: Normal: 0*4kB 2*8kB 13*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3712kB
Feb 18 15:05:54 fox kernel: HighMem: 58*4kB 19*8kB 34*16kB 43*32kB 28*64kB 15542*128kB 19468*256kB 8287*512kB 2436*1024kB 675*2048kB 5*4096kB = 15117568kB
Feb 18 15:05:54 fox kernel: Swap cache: add 22177, delete 21910, find 7904/9381, race 0+0
Feb 18 15:05:54 fox kernel: 0 bounce buffer pages
Feb 18 15:05:54 fox kernel: Free swap:       2093728kB
Feb 18 15:05:55 fox kernel: 17301504 pages of RAM
Feb 18 15:05:55 fox kernel: 16547690 pages of HIGHMEM
Feb 18 15:05:55 fox kernel: 661425 reserved pages
Feb 18 15:05:55 fox kernel: 3077721 pages shared
Feb 18 15:05:55 fox kernel: 267 pages swap cached
Feb 18 15:05:55 fox kernel: Out of Memory: Killed process 20210 (winmed).
Feb 18 15:05:55 fox kernel: Fixed up OOM kill of mm-less task
 
Here is /proc/meminfo when the server is running normal:
MemTotal:     66560316 kB
MemFree:      44910252 kB
Buffers:         93112 kB
Cached:       16412208 kB
SwapCached:          0 kB
Active:        9201024 kB
Inactive:     12076300 kB
HighTotal:    66190760 kB
HighFree:     44888192 kB
LowTotal:       369556 kB
LowFree:         22060 kB
SwapTotal:     2096472 kB
SwapFree:      2096472 kB
Dirty:           30476 kB
Writeback:           0 kB
Mapped:        4815880 kB
Slab:           234368 kB
CommitLimit:  35376628 kB
Committed_AS: 13765424 kB
PageTables:     113736 kB
VmallocTotal:   106488 kB
VmallocUsed:      2704 kB
VmallocChunk:   103460 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

[+][-]02/19/08 02:49 PM, ID: 20933397Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]02/20/08 12:23 AM, ID: 20935913Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]02/20/08 03:42 AM, ID: 20936788Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]02/20/08 04:01 AM, ID: 20936867Accepted Solution

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

About this solution

Zones: Linux Administration, Red Hat Linux
Tags: oom, killer
Sign Up Now!
Solution Provided By: ravenpl
Participating Experts: 1
Solution Grade: A
 
 
Loading Advertisement...
20091021-EE-VQP-81 / EE_QW_2_20070628