LINIUX OOM Killer

My Linus server rebooted because of low availiable memory. The OOM killed many processes and the server had to be rebooted.

What caused the server to reboot?


Please see log:

Oct  9 04:01:03 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:01:03 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:01:03 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: Normal per-cpu:
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem per-cpu:
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel:
Oct  9 04:01:04 jmdcprodmq2 kernel: Free pages:      292012kB (278784kB HighMem)
Oct  9 04:01:04 jmdcprodmq2 kernel: Active:110968 inactive:370406 dirty:8216 writeback:1 unstable:0 free:73003 slab:214643 mapped:110430 pagetables:3626
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4684 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal free:672kB min:928kB low:1856kB high:2784kB active:340kB inactive:148kB present:901120kB pages_scanned:923 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem free:278784kB min:512kB low:1024kB high:1536kB active:443532kB inactive:1481476kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal: 0*4kB 0*8kB 16*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 672kB
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem: 458*4kB 47*8kB 56*16kB 29*32kB 83*64kB 67*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 278784kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:01:04 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:01:04 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:01:04 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:01:04 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:01:04 jmdcprodmq2 kernel: 409632 pages shared
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 pages swap cached
Oct  9 04:01:04 jmdcprodmq2 kernel: Out of Memory: Killed process 13829 (amqzlaa0).
Oct  9 04:01:10 jmdcprodmq2 MQSeries: FFST (34,5,35,536891401) failed: /var/mqm/errors/AMQ27941.0.FDC (errno=0)
Oct  9 04:01:10 jmdcprodmq2 MQSeries: FFST record created in /var/mqm/errors/AMQ27941.0.FDC
Oct  9 04:01:30 jmdcprodmq2 clurgmgrd: [4645]: <info> Executing /opt/MQHA1/JMDCQMGR/scripts/MQ_HA status
Oct  9 04:01:30 jmdcprodmq2 su(pam_unix)[11651]: session opened for user mqm by (uid=0)
Oct  9 04:01:31 jmdcprodmq2 su(pam_unix)[11651]: session closed for user mqm
Oct  9 04:02:00 jmdcprodmq2 clurgmgrd: [4645]: <info> Executing /opt/MQHA1/JMDCQMGR/scripts/MQ_HA status
Oct  9 04:02:00 jmdcprodmq2 su(pam_unix)[11849]: session opened for user mqm by (uid=0)
Oct  9 04:02:00 jmdcprodmq2 su(pam_unix)[11849]: session closed for user mqm
Oct  9 04:02:02 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:02:02 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:02:02 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: Normal per-cpu:
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: HighMem per-cpu:
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 MQSeries: FFST (34,5,35,536891401) failed: /var/mqm/errors/AMQ27941.0.FDC (errno=0)
Oct  9 04:02:11 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:13 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:13 jmdcprodmq2 MQSeries: FFST record created in /var/mqm/errors/AMQ27941.0.FDC
Oct  9 04:02:13 jmdcprodmq2 kernel:
Oct  9 04:02:13 jmdcprodmq2 kernel: Free pages:      283540kB (270080kB HighMem)
Oct  9 04:02:15 jmdcprodmq2 kernel: Active:112819 inactive:370363 dirty:2153 writeback:0 unstable:0 free:70885 slab:214528 mapped:112229 pagetables:4018
Oct  9 04:02:15 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4921 all_unreclaimable? yes
Oct  9 04:02:15 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:20 jmdcprodmq2 kernel: Normal free:904kB min:928kB low:1856kB high:2784kB active:488kB inactive:0kB present:901120kB pages_scanned:517 all_unreclaimable? yes
Oct  9 04:02:20 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:20 jmdcprodmq2 kernel: HighMem free:270080kB min:512kB low:1024kB high:1536kB active:450788kB inactive:1481452kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:02:20 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:20 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:02:20 jmdcprodmq2 kernel: Normal: 0*4kB 25*8kB 18*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904kB
Oct  9 04:02:25 jmdcprodmq2 kernel: HighMem: 320*4kB 106*8kB 33*16kB 19*32kB 33*64kB 30*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 270080kB
Oct  9 04:02:25 jmdcprodmq2 MQSeries: FFST (34,5,35,536891401) failed: /var/mqm/errors/AMQ27941.0.FDC (errno=0)
Oct  9 04:02:25 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:02:25 jmdcprodmq2 MQSeries: FFST record created in /var/mqm/errors/AMQ27941.0.FDC
Oct  9 04:02:25 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:02:25 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:02:25 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:02:25 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:02:25 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:02:25 jmdcprodmq2 kernel: 417368 pages shared
Oct  9 04:02:26 jmdcprodmq2 kernel: 0 pages swap cached
Oct  9 04:02:26 jmdcprodmq2 kernel: Out of Memory: Killed process 11477 (amqzlaa0).
Oct  9 04:02:26 jmdcprodmq2 kernel: Fixed up OOM kill of mm-less task
Oct  9 04:02:26 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:02:26 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:02:26 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: Normal per-cpu:
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: HighMem per-cpu:
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:27 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:27 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel:
Oct  9 04:02:34 jmdcprodmq2 kernel: Free pages:      284308kB (270848kB HighMem)
Oct  9 04:02:34 jmdcprodmq2 clurgmgrd: [4645]: <info> Executing /opt/MQHA1/JMDCQMGR/scripts/MQ_HA status
Oct  9 04:02:34 jmdcprodmq2 kernel: Active:112603 inactive:370393 dirty:2154 writeback:0 unstable:0 free:71077 slab:214536 mapped:112034 pagetables:3971
Oct  9 04:02:34 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4926 all_unreclaimable? yes
Oct  9 04:02:34 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:34 jmdcprodmq2 kernel: Normal free:904kB min:928kB low:1856kB high:2784kB active:352kB inactive:120kB present:901120kB pages_scanned:38137 all_unreclaimable? yes
Oct  9 04:02:34 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:34 jmdcprodmq2 kernel: HighMem free:270848kB min:512kB low:1024kB high:1536kB active:450060kB inactive:1481452kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:02:34 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:35 jmdcprodmq2 su(pam_unix)[12398]: session opened for user mqm by (uid=0)
Oct  9 04:02:35 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:02:44 jmdcprodmq2 kernel: Normal: 0*4kB 25*8kB 18*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904kB
Oct  9 04:02:44 jmdcprodmq2 kernel: HighMem: 348*4kB 126*8kB 52*16kB 23*32kB 34*64kB 30*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 270848kB
Oct  9 04:02:44 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:02:44 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:02:44 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:02:44 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:02:44 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:02:44 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:02:44 jmdcprodmq2 kernel: 416197 pages shared
Oct  9 04:02:44 jmdcprodmq2 kernel: 0 pages swap cached
Oct  9 04:02:44 jmdcprodmq2 kernel: Out of Memory: Killed process 19528 (amqzlaa0).
Oct  9 04:02:44 jmdcprodmq2 kernel: Fixed up OOM kill of mm-less task
Oct  9 04:02:45 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:02:45 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:02:45 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: Normal per-cpu:
rbttAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

woolmilkporcCommented:
Hi,
have a look at these links:


http://linux-mm.org/OOM_Killer
http://en.wikipedia.org/wiki/Out_of_memory

Basically, the oom-killer (out-of-memory killer) tries to free up memory by killing processes.
You'll have to investigate which prodess(es) eat up your memory.

wmp


0
woolmilkporcCommented:
0
rbttAuthor Commented:
What I am tying to find is if the Server ran out of High or Low memory.
0
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

woolmilkporcCommented:
Oct  9 04:02:15 jmdcprodmq2 kernel: DMA free:12556kB
Oct  9 04:02:34 jmdcprodmq2 kernel: Normal free:672kB
Oct  9 04:02:34 jmdcprodmq2 kernel: HighMem free:278784kB


Since your zone_normal is rather full, but there's still a bit HighMem available, and since zone_normal is used to map pages of high memory into, I guess it's not few big processes that consume the memory, but a lot of relatively small ones.
So, your problem seems to be at first place in low memory, but since there is not too much high memory left, you also have an overall memory problem.
Note: There is no possibility to increase the size of zone_normal, since it can only go from 16MB to 896 MB.

At this first look, it's a process (application?) problem. I would bet increasing physical memory wouldn't have too much effect.

wmp

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
rbttAuthor Commented:
Can anyone summarize what is happening with the extract below? Is it correct to assume that the system ran out of normal memory?

What would cause normal memory to run out?




Oct  9 04:01:04 jmdcprodmq2 kernel: Free pages:      292012kB (278784kB HighMem)
Oct  9 04:01:04 jmdcprodmq2 kernel: Active:110968 inactive:370406 dirty:8216 writeback:1 unstable:0 free:73003 slab:214643 mapped:110430 pagetables:3626
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4684 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal free:672kB min:928kB low:1856kB high:2784kB active:340kB inactive:148kB present:901120kB pages_scanned:923 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem free:278784kB min:512kB low:1024kB high:1536kB active:443532kB inactive:1481476kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal: 0*4kB 0*8kB 16*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 672kB
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem: 458*4kB 47*8kB 56*16kB 29*32kB 83*64kB 67*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 278784kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:01:04 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:01:04 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:01:04 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:01:04 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:01:04 jmdcprodmq2 kernel: 409632 pages shared
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 pages swap cached
0
gheistCommented:
Since there is much of free space available you just overkilled with mlocked shared memory.

409632 pages shared ~ 1,5GB ram is locked by database or something like that, that could have exhausted lowmem.

Oracle?

What does uptime and vmstat 5 5 show in session you start when everything goes down?
Are you disks OK by means of S.M.A.R.T?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.