sel999
asked on
Postgres 8.1.2 crashed now how to figure out why?
Hi, I have a production PostgreSQL database used by a 24/7 application. Â I have it running on RHEL 4.2 server running PostgreSQL 8.1.2 on SUN v20Z servers &Â thankfully running RH Cluster server (helpped me out today :) Â This morning it decided to crash but was brought back online by the cluster software within a matter of seconds. The thing is I am left with the situation of trying to work out why it happend. It appeared to run out of memory (Out of Memory: Killed process 22431 (postmaster).
) so could be a Postgres bug. Does any one have idea's of where to start? Â Below is the output during the incident logged in the /var/log/messages file
Thanks,
Sel
Apr 18 11:46:02 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:46:02 node1 su(pam_unix)[8178]: session opened for user postgres by (uid=0)
Apr 18 11:46:03 node1 su(pam_unix)[8178]: session closed for user postgres
Apr 18 11:48:24 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:24 node1 kernel: Mem-info:
Apr 18 11:48:24 node1 kernel: DMA per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:24 node1 kernel: Normal per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:24 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:24 node1 kernel:
Apr 18 11:48:24 node1 kernel: Free pages: Â Â Â 13004kB (0kB HighMem)
Apr 18 11:48:24 node1 kernel: Active:225429 inactive:223531 dirty:0 writeback:0 unstable:0 free:3251 slab:49026 mapped:446637 pagetables:6684
Apr 18 11:48:24 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:48:26 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:901716kB inactive:894124kB present:2080192kB pages_scanned:2154735 all_unreclaimable? yes
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:48:29 node1 kernel: Normal: 10*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB <7>Losing some ticks... checking if CPU frequency changed.
Apr 18 11:48:29 node1 kernel: = 1408kB
Apr 18 11:48:31 node1 kernel: HighMem: empty
Apr 18 11:48:34 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:48:34 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:48:34 node1 kernel: 524144 pages of RAM
Apr 18 11:48:34 node1 kernel: 10384 reserved pages
Apr 18 11:48:34 node1 kernel: 4113 pages shared
Apr 18 11:48:34 node1 kernel: 0 pages swap cached
Apr 18 11:48:34 node1 kernel: Out of Memory: Killed process 22429 (postmaster).
Apr 18 11:48:34 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:34 node1 kernel: Mem-info:
Apr 18 11:48:36 node1 kernel: DMA per-cpu:
Apr 18 11:48:39 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:45 node1 kernel: Normal per-cpu:
Apr 18 11:48:45 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:45 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:46 node1 kernel:
Apr 18 11:48:49 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:48:52 node1 kernel: Active:192022 inactive:257223 dirty:0 writeback:0 unstable:0 free:3249 slab:48954 mapped:446565 pagetables:6658
Apr 18 11:49:10 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:50:00 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:02 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:768088kB inactive:1028892kB present:2080192kB pages_scanned:3202287 all_unreclaimable? yes
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:26 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:28 node1 kernel: HighMem: empty
Apr 18 11:51:29 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:30 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:30 node1 kernel: 524144 pages of RAM
Apr 18 11:51:31 node1 kernel: 10384 reserved pages
Apr 18 11:51:31 node1 kernel: 3216 pages shared
Apr 18 11:51:31 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:51:31 node1 su[7499]: Warning! Â Could not relabel /dev/pts/1 with user_u:object_r:initrc_dev pts_t, not relabeling.
Apr 18 11:51:31 node1 su[9596]: Warning! Â Could not relabel /dev/pts/3 with user_u:object_r:initrc_dev pts_t, not relabeling.
Apr 18 11:51:31 node1 su[17331]: Warning! Â Could not relabel /dev/pts/7 with user_u:object_r:initrc_dev pts_t, not relabeling.
Apr 18 11:51:31 node1 su[13172]: Warning! Â Could not relabel /dev/pts/4 with user_u:object_r:initrc_dev pts_t, not relabeling.
Apr 18 11:51:32 node1 kernel: 0 pages swap cached
Apr 18 11:51:32 node1 su(pam_unix)[7499]: session closed for user root
Apr 18 11:51:32 node1 su(pam_unix)[9596]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[17331]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[13172]: session closed for user root
Apr 18 11:51:33 node1 kernel: Out of Memory: Killed process 22424 (postmaster).
Apr 18 11:51:33 node1 su(pam_unix)[8339]: session opened for user postgres by (uid=0)
Apr 18 11:51:33 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:34 node1 kernel: Mem-info:
Apr 18 11:51:34 node1 kernel: DMA per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:34 node1 kernel: Normal per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:34 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:34 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:234490 inactive:215587 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:937960kB inactive:862348kB present:2080192kB pages_scanned:5615412 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22430 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:35 node1 kernel: Normal per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:35 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:35 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:256856 inactive:193221 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:1027424kB inactive:772884kB present:2080192kB pages_scanned:6180966 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22431 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 su(pam_unix)[8339]: session closed for user postgres
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> status on script "Postgress App" returned 1 (generic error)
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:36 node1 kernel: Normal per-cpu:
Apr 18 11:51:36 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:36 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:36 node1 kernel:
Apr 18 11:51:36 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:51:36 node1 kernel: Active:224225 inactive:224642 dirty:0 writeback:0 unstable:0 free:3249 slab:48923 mapped:446679 pagetables:6589
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> Stopping service Postgres Database Cluster
Apr 18 11:51:36 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:896900kB inactive:898568kB present:2080192kB pages_scanned:8011971 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:36 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh stop
Apr 18 11:51:36 node1 kernel: Normal: 0*4kB 15*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session opened for user postgres by (uid=0)
Apr 18 11:51:36 node1 kernel: HighMem: empty
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session closed for user postgres
Apr 18 11:51:37 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:37 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:37 node1 kernel: 524144 pages of RAM
Apr 18 11:51:37 node1 kernel: 10384 reserved pages
Apr 18 11:51:37 node1 kernel: 3220 pages shared
Apr 18 11:51:37 node1 kernel: 0 pages swap cached
Apr 18 11:51:37 node1 kernel: Out of Memory: Killed process 4092 (gnome-terminal).
Apr 18 11:51:37 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:37 node1 kernel: Mem-info:
Apr 18 11:51:37 node1 clurgmgrd: [15304]: <info> Removing IPv4 address 10.2.33.45 from eth0
Apr 18 11:51:37 node1 kernel: DMA per-cpu:
Apr 18 11:51:37 node1 gconfd (sels-8420): starting (version 2.8.1), pid 8420 user 'sels'
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:37 node1 kernel: Normal per-cpu:
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/g conf.xml.m andatory" to a read-only configuration source at position 0
Apr 18 11:51:37 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readwrite:/home/sels/ .gconf" to a writable configuration source at position 1
Apr 18 11:51:38 node1 kernel:
Apr 18 11:51:38 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/g conf.xml.d efaults" to a read-only configuration source at position 2
Apr 18 11:51:38 node1 kernel: Free pages: Â Â Â 13004kB (0kB HighMem)
Apr 18 11:51:38 node1 kernel: Active:225234 inactive:223793 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:38 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:900936kB inactive:895172kB present:2080192kB pages_scanned:3741771 all_unreclaimable? no
Apr 18 11:51:39 node1 sshd(pam_unix)[8479]: session opened for user root by (uid=0)
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436843, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 3060 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 2927 (nautilus).
Apr 18 11:51:40 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:40 node1 kernel: Mem-info:
Apr 18 11:51:40 node1 kernel: DMA per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:40 node1 kernel: Normal per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:40 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:40 node1 kernel:
Apr 18 11:51:40 node1 kernel: Free pages: Â Â Â 13004kB (0kB HighMem)
Apr 18 11:51:40 node1 kernel: Active:239766 inactive:209805 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:40 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:959064kB inactive:839220kB present:2080192kB pages_scanned:7270296 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436847, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 2516 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 3074 (nautilus).
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <notice> Forcefully unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22433 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22434 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22435 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:47 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster is recovering
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Recovering failed service Postgres Database Cluster
Apr 18 11:51:48 node1 kernel: kjournald starting. Â Commit interval 5 seconds
Apr 18 11:51:48 node1 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Apr 18 11:51:48 node1 kernel: EXT3 FS on sdb1, internal journal
Apr 18 11:51:48 node1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Apr 18 11:51:48 node1 kernel: SELinux: initialized (dev sdb1, type ext3), uses xattr
Apr 18 11:51:49 node1 clurgmgrd: [15304]: <info> Adding IPv4 address 10.2.33.45 to eth0
Apr 18 11:51:50 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh start
Apr 18 11:51:50 node1 su(pam_unix)[8713]: session opened for user postgres by (uid=0)
Apr 18 11:51:51 node1 su(pam_unix)[8713]: session closed for user postgres
Apr 18 11:51:51 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster started
) so could be a Postgres bug. Does any one have idea's of where to start? Â Below is the output during the incident logged in the /var/log/messages file
Thanks,
Sel
Apr 18 11:46:02 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:46:02 node1 su(pam_unix)[8178]: session opened for user postgres by (uid=0)
Apr 18 11:46:03 node1 su(pam_unix)[8178]: session closed for user postgres
Apr 18 11:48:24 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:24 node1 kernel: Mem-info:
Apr 18 11:48:24 node1 kernel: DMA per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:24 node1 kernel: Normal per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:24 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:24 node1 kernel:
Apr 18 11:48:24 node1 kernel: Free pages: Â Â Â 13004kB (0kB HighMem)
Apr 18 11:48:24 node1 kernel: Active:225429 inactive:223531 dirty:0 writeback:0 unstable:0 free:3251 slab:49026 mapped:446637 pagetables:6684
Apr 18 11:48:24 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:48:26 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:901716kB inactive:894124kB present:2080192kB pages_scanned:2154735 all_unreclaimable? yes
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:48:29 node1 kernel: Normal: 10*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB <7>Losing some ticks... checking if CPU frequency changed.
Apr 18 11:48:29 node1 kernel: = 1408kB
Apr 18 11:48:31 node1 kernel: HighMem: empty
Apr 18 11:48:34 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:48:34 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:48:34 node1 kernel: 524144 pages of RAM
Apr 18 11:48:34 node1 kernel: 10384 reserved pages
Apr 18 11:48:34 node1 kernel: 4113 pages shared
Apr 18 11:48:34 node1 kernel: 0 pages swap cached
Apr 18 11:48:34 node1 kernel: Out of Memory: Killed process 22429 (postmaster).
Apr 18 11:48:34 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:34 node1 kernel: Mem-info:
Apr 18 11:48:36 node1 kernel: DMA per-cpu:
Apr 18 11:48:39 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:45 node1 kernel: Normal per-cpu:
Apr 18 11:48:45 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:45 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:46 node1 kernel:
Apr 18 11:48:49 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:48:52 node1 kernel: Active:192022 inactive:257223 dirty:0 writeback:0 unstable:0 free:3249 slab:48954 mapped:446565 pagetables:6658
Apr 18 11:49:10 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:50:00 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:02 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:768088kB inactive:1028892kB present:2080192kB pages_scanned:3202287 all_unreclaimable? yes
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:26 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:28 node1 kernel: HighMem: empty
Apr 18 11:51:29 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:30 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:30 node1 kernel: 524144 pages of RAM
Apr 18 11:51:31 node1 kernel: 10384 reserved pages
Apr 18 11:51:31 node1 kernel: 3216 pages shared
Apr 18 11:51:31 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:51:31 node1 su[7499]: Warning! Â Could not relabel /dev/pts/1 with user_u:object_r:initrc_dev
Apr 18 11:51:31 node1 su[9596]: Warning! Â Could not relabel /dev/pts/3 with user_u:object_r:initrc_dev
Apr 18 11:51:31 node1 su[17331]: Warning! Â Could not relabel /dev/pts/7 with user_u:object_r:initrc_dev
Apr 18 11:51:31 node1 su[13172]: Warning! Â Could not relabel /dev/pts/4 with user_u:object_r:initrc_dev
Apr 18 11:51:32 node1 kernel: 0 pages swap cached
Apr 18 11:51:32 node1 su(pam_unix)[7499]: session closed for user root
Apr 18 11:51:32 node1 su(pam_unix)[9596]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[17331]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[13172]: session closed for user root
Apr 18 11:51:33 node1 kernel: Out of Memory: Killed process 22424 (postmaster).
Apr 18 11:51:33 node1 su(pam_unix)[8339]: session opened for user postgres by (uid=0)
Apr 18 11:51:33 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:34 node1 kernel: Mem-info:
Apr 18 11:51:34 node1 kernel: DMA per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:34 node1 kernel: Normal per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:34 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:34 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:234490 inactive:215587 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:937960kB inactive:862348kB present:2080192kB pages_scanned:5615412 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22430 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:35 node1 kernel: Normal per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:35 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:35 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:256856 inactive:193221 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:1027424kB inactive:772884kB present:2080192kB pages_scanned:6180966 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22431 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 su(pam_unix)[8339]: session closed for user postgres
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> status on script "Postgress App" returned 1 (generic error)
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:36 node1 kernel: Normal per-cpu:
Apr 18 11:51:36 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:36 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:36 node1 kernel:
Apr 18 11:51:36 node1 kernel: Free pages: Â Â Â 12996kB (0kB HighMem)
Apr 18 11:51:36 node1 kernel: Active:224225 inactive:224642 dirty:0 writeback:0 unstable:0 free:3249 slab:48923 mapped:446679 pagetables:6589
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> Stopping service Postgres Database Cluster
Apr 18 11:51:36 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:896900kB inactive:898568kB present:2080192kB pages_scanned:8011971 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:36 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh stop
Apr 18 11:51:36 node1 kernel: Normal: 0*4kB 15*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session opened for user postgres by (uid=0)
Apr 18 11:51:36 node1 kernel: HighMem: empty
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session closed for user postgres
Apr 18 11:51:37 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:37 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:37 node1 kernel: 524144 pages of RAM
Apr 18 11:51:37 node1 kernel: 10384 reserved pages
Apr 18 11:51:37 node1 kernel: 3220 pages shared
Apr 18 11:51:37 node1 kernel: 0 pages swap cached
Apr 18 11:51:37 node1 kernel: Out of Memory: Killed process 4092 (gnome-terminal).
Apr 18 11:51:37 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:37 node1 kernel: Mem-info:
Apr 18 11:51:37 node1 clurgmgrd: [15304]: <info> Removing IPv4 address 10.2.33.45 from eth0
Apr 18 11:51:37 node1 kernel: DMA per-cpu:
Apr 18 11:51:37 node1 gconfd (sels-8420): starting (version 2.8.1), pid 8420 user 'sels'
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:37 node1 kernel: Normal per-cpu:
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/g
Apr 18 11:51:37 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readwrite:/home/sels/
Apr 18 11:51:38 node1 kernel:
Apr 18 11:51:38 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/g
Apr 18 11:51:38 node1 kernel: Free pages: Â Â Â 13004kB (0kB HighMem)
Apr 18 11:51:38 node1 kernel: Active:225234 inactive:223793 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:38 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:900936kB inactive:895172kB present:2080192kB pages_scanned:3741771 all_unreclaimable? no
Apr 18 11:51:39 node1 sshd(pam_unix)[8479]: session opened for user root by (uid=0)
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436843, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 3060 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 2927 (nautilus).
Apr 18 11:51:40 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:40 node1 kernel: Mem-info:
Apr 18 11:51:40 node1 kernel: DMA per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:40 node1 kernel: Normal per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:40 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:40 node1 kernel:
Apr 18 11:51:40 node1 kernel: Free pages: Â Â Â 13004kB (0kB HighMem)
Apr 18 11:51:40 node1 kernel: Active:239766 inactive:209805 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:40 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:959064kB inactive:839220kB present:2080192kB pages_scanned:7270296 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436847, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap: Â Â Â Â Â Â 0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 2516 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 3074 (nautilus).
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <notice> Forcefully unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22433 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22434 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22435 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:47 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster is recovering
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Recovering failed service Postgres Database Cluster
Apr 18 11:51:48 node1 kernel: kjournald starting. Â Commit interval 5 seconds
Apr 18 11:51:48 node1 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Apr 18 11:51:48 node1 kernel: EXT3 FS on sdb1, internal journal
Apr 18 11:51:48 node1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Apr 18 11:51:48 node1 kernel: SELinux: initialized (dev sdb1, type ext3), uses xattr
Apr 18 11:51:49 node1 clurgmgrd: [15304]: <info> Adding IPv4 address 10.2.33.45 to eth0
Apr 18 11:51:50 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh start
Apr 18 11:51:50 node1 su(pam_unix)[8713]: session opened for user postgres by (uid=0)
Apr 18 11:51:51 node1 su(pam_unix)[8713]: session closed for user postgres
Apr 18 11:51:51 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster started
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
What is so incomplete in my request for additional system information ???
ASKER
Hiya, not sure what you mean?
Why B grade ???
ASKER
Oh I don't actaully remember setting a grade. I'm sorry I would have given a A grade but don't remember choosing a grade at all. I will see if I can change it. Rgs, Sel
There is also a possibility that you are running too many child processes with Postgres and they run out of memory.