• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1110
  • Last Modified:

LINIUX OOM Killer

My Linus server rebooted because of low availiable memory. The OOM killed many processes and the server had to be rebooted.

What caused the server to reboot?


Please see log:

Oct  9 04:01:03 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:01:03 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:01:03 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:01:03 jmdcprodmq2 kernel: Normal per-cpu:
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:01:03 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem per-cpu:
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:01:04 jmdcprodmq2 kernel:
Oct  9 04:01:04 jmdcprodmq2 kernel: Free pages:      292012kB (278784kB HighMem)
Oct  9 04:01:04 jmdcprodmq2 kernel: Active:110968 inactive:370406 dirty:8216 writeback:1 unstable:0 free:73003 slab:214643 mapped:110430 pagetables:3626
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4684 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal free:672kB min:928kB low:1856kB high:2784kB active:340kB inactive:148kB present:901120kB pages_scanned:923 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem free:278784kB min:512kB low:1024kB high:1536kB active:443532kB inactive:1481476kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal: 0*4kB 0*8kB 16*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 672kB
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem: 458*4kB 47*8kB 56*16kB 29*32kB 83*64kB 67*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 278784kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:01:04 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:01:04 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:01:04 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:01:04 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:01:04 jmdcprodmq2 kernel: 409632 pages shared
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 pages swap cached
Oct  9 04:01:04 jmdcprodmq2 kernel: Out of Memory: Killed process 13829 (amqzlaa0).
Oct  9 04:01:10 jmdcprodmq2 MQSeries: FFST (34,5,35,536891401) failed: /var/mqm/errors/AMQ27941.0.FDC (errno=0)
Oct  9 04:01:10 jmdcprodmq2 MQSeries: FFST record created in /var/mqm/errors/AMQ27941.0.FDC
Oct  9 04:01:30 jmdcprodmq2 clurgmgrd: [4645]: <info> Executing /opt/MQHA1/JMDCQMGR/scripts/MQ_HA status
Oct  9 04:01:30 jmdcprodmq2 su(pam_unix)[11651]: session opened for user mqm by (uid=0)
Oct  9 04:01:31 jmdcprodmq2 su(pam_unix)[11651]: session closed for user mqm
Oct  9 04:02:00 jmdcprodmq2 clurgmgrd: [4645]: <info> Executing /opt/MQHA1/JMDCQMGR/scripts/MQ_HA status
Oct  9 04:02:00 jmdcprodmq2 su(pam_unix)[11849]: session opened for user mqm by (uid=0)
Oct  9 04:02:00 jmdcprodmq2 su(pam_unix)[11849]: session closed for user mqm
Oct  9 04:02:02 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:02:02 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:02:02 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:02:02 jmdcprodmq2 kernel: Normal per-cpu:
Oct  9 04:02:02 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: HighMem per-cpu:
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:10 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:10 jmdcprodmq2 MQSeries: FFST (34,5,35,536891401) failed: /var/mqm/errors/AMQ27941.0.FDC (errno=0)
Oct  9 04:02:11 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:13 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:13 jmdcprodmq2 MQSeries: FFST record created in /var/mqm/errors/AMQ27941.0.FDC
Oct  9 04:02:13 jmdcprodmq2 kernel:
Oct  9 04:02:13 jmdcprodmq2 kernel: Free pages:      283540kB (270080kB HighMem)
Oct  9 04:02:15 jmdcprodmq2 kernel: Active:112819 inactive:370363 dirty:2153 writeback:0 unstable:0 free:70885 slab:214528 mapped:112229 pagetables:4018
Oct  9 04:02:15 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4921 all_unreclaimable? yes
Oct  9 04:02:15 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:20 jmdcprodmq2 kernel: Normal free:904kB min:928kB low:1856kB high:2784kB active:488kB inactive:0kB present:901120kB pages_scanned:517 all_unreclaimable? yes
Oct  9 04:02:20 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:20 jmdcprodmq2 kernel: HighMem free:270080kB min:512kB low:1024kB high:1536kB active:450788kB inactive:1481452kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:02:20 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:20 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:02:20 jmdcprodmq2 kernel: Normal: 0*4kB 25*8kB 18*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904kB
Oct  9 04:02:25 jmdcprodmq2 kernel: HighMem: 320*4kB 106*8kB 33*16kB 19*32kB 33*64kB 30*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 270080kB
Oct  9 04:02:25 jmdcprodmq2 MQSeries: FFST (34,5,35,536891401) failed: /var/mqm/errors/AMQ27941.0.FDC (errno=0)
Oct  9 04:02:25 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:02:25 jmdcprodmq2 MQSeries: FFST record created in /var/mqm/errors/AMQ27941.0.FDC
Oct  9 04:02:25 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:02:25 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:02:25 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:02:25 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:02:25 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:02:25 jmdcprodmq2 kernel: 417368 pages shared
Oct  9 04:02:26 jmdcprodmq2 kernel: 0 pages swap cached
Oct  9 04:02:26 jmdcprodmq2 kernel: Out of Memory: Killed process 11477 (amqzlaa0).
Oct  9 04:02:26 jmdcprodmq2 kernel: Fixed up OOM kill of mm-less task
Oct  9 04:02:26 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:02:26 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:02:26 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:02:26 jmdcprodmq2 kernel: Normal per-cpu:
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: HighMem per-cpu:
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct  9 04:02:26 jmdcprodmq2 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct  9 04:02:27 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct  9 04:02:27 jmdcprodmq2 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct  9 04:02:34 jmdcprodmq2 kernel:
Oct  9 04:02:34 jmdcprodmq2 kernel: Free pages:      284308kB (270848kB HighMem)
Oct  9 04:02:34 jmdcprodmq2 clurgmgrd: [4645]: <info> Executing /opt/MQHA1/JMDCQMGR/scripts/MQ_HA status
Oct  9 04:02:34 jmdcprodmq2 kernel: Active:112603 inactive:370393 dirty:2154 writeback:0 unstable:0 free:71077 slab:214536 mapped:112034 pagetables:3971
Oct  9 04:02:34 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4926 all_unreclaimable? yes
Oct  9 04:02:34 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:34 jmdcprodmq2 kernel: Normal free:904kB min:928kB low:1856kB high:2784kB active:352kB inactive:120kB present:901120kB pages_scanned:38137 all_unreclaimable? yes
Oct  9 04:02:34 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:34 jmdcprodmq2 kernel: HighMem free:270848kB min:512kB low:1024kB high:1536kB active:450060kB inactive:1481452kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:02:34 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:02:35 jmdcprodmq2 su(pam_unix)[12398]: session opened for user mqm by (uid=0)
Oct  9 04:02:35 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:02:44 jmdcprodmq2 kernel: Normal: 0*4kB 25*8kB 18*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904kB
Oct  9 04:02:44 jmdcprodmq2 kernel: HighMem: 348*4kB 126*8kB 52*16kB 23*32kB 34*64kB 30*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 270848kB
Oct  9 04:02:44 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:02:44 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:02:44 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:02:44 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:02:44 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:02:44 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:02:44 jmdcprodmq2 kernel: 416197 pages shared
Oct  9 04:02:44 jmdcprodmq2 kernel: 0 pages swap cached
Oct  9 04:02:44 jmdcprodmq2 kernel: Out of Memory: Killed process 19528 (amqzlaa0).
Oct  9 04:02:44 jmdcprodmq2 kernel: Fixed up OOM kill of mm-less task
Oct  9 04:02:45 jmdcprodmq2 kernel: oom-killer: gfp_mask=0xd0
Oct  9 04:02:45 jmdcprodmq2 kernel: Mem-info:
Oct  9 04:02:45 jmdcprodmq2 kernel: DMA per-cpu:
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct  9 04:02:45 jmdcprodmq2 kernel: Normal per-cpu:
0
rbtt
Asked:
rbtt
2 Solutions
 
woolmilkporcCommented:
Hi,
have a look at these links:


http://linux-mm.org/OOM_Killer
http://en.wikipedia.org/wiki/Out_of_memory

Basically, the oom-killer (out-of-memory killer) tries to free up memory by killing processes.
You'll have to investigate which prodess(es) eat up your memory.

wmp


0
 
woolmilkporcCommented:
0
 
rbttAuthor Commented:
What I am tying to find is if the Server ran out of High or Low memory.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
woolmilkporcCommented:
Oct  9 04:02:15 jmdcprodmq2 kernel: DMA free:12556kB
Oct  9 04:02:34 jmdcprodmq2 kernel: Normal free:672kB
Oct  9 04:02:34 jmdcprodmq2 kernel: HighMem free:278784kB


Since your zone_normal is rather full, but there's still a bit HighMem available, and since zone_normal is used to map pages of high memory into, I guess it's not few big processes that consume the memory, but a lot of relatively small ones.
So, your problem seems to be at first place in low memory, but since there is not too much high memory left, you also have an overall memory problem.
Note: There is no possibility to increase the size of zone_normal, since it can only go from 16MB to 896 MB.

At this first look, it's a process (application?) problem. I would bet increasing physical memory wouldn't have too much effect.

wmp

0
 
rbttAuthor Commented:
Can anyone summarize what is happening with the extract below? Is it correct to assume that the system ran out of normal memory?

What would cause normal memory to run out?




Oct  9 04:01:04 jmdcprodmq2 kernel: Free pages:      292012kB (278784kB HighMem)
Oct  9 04:01:04 jmdcprodmq2 kernel: Active:110968 inactive:370406 dirty:8216 writeback:1 unstable:0 free:73003 slab:214643 mapped:110430 pagetables:3626
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA free:12556kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:4684 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal free:672kB min:928kB low:1856kB high:2784kB active:340kB inactive:148kB present:901120kB pages_scanned:923 all_unreclaimable? yes
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem free:278784kB min:512kB low:1024kB high:1536kB active:443532kB inactive:1481476kB present:2227952kB pages_scanned:0 all_unreclaimable? no
Oct  9 04:01:04 jmdcprodmq2 kernel: protections[]: 0 0 0
Oct  9 04:01:04 jmdcprodmq2 kernel: DMA: 1*4kB 5*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12556kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Normal: 0*4kB 0*8kB 16*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 672kB
Oct  9 04:01:04 jmdcprodmq2 kernel: HighMem: 458*4kB 47*8kB 56*16kB 29*32kB 83*64kB 67*128kB 1*256kB 39*512kB 25*1024kB 5*2048kB 50*4096kB = 278784kB
Oct  9 04:01:04 jmdcprodmq2 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 bounce buffer pages
Oct  9 04:01:04 jmdcprodmq2 kernel: Free swap:       4194296kB
Oct  9 04:01:04 jmdcprodmq2 kernel: 786364 pages of RAM
Oct  9 04:01:04 jmdcprodmq2 kernel: 556988 pages of HIGHMEM
Oct  9 04:01:04 jmdcprodmq2 kernel: 7606 reserved pages
Oct  9 04:01:04 jmdcprodmq2 kernel: 409632 pages shared
Oct  9 04:01:04 jmdcprodmq2 kernel: 0 pages swap cached
0
 
gheistCommented:
Since there is much of free space available you just overkilled with mlocked shared memory.

409632 pages shared ~ 1,5GB ram is locked by database or something like that, that could have exhausted lowmem.

Oracle?

What does uptime and vmstat 5 5 show in session you start when everything goes down?
Are you disks OK by means of S.M.A.R.T?
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now