?
Solved

Postgres 8.1.2 crashed now how to figure out why?

Posted on 2006-04-18
6
Medium Priority
?
556 Views
Last Modified: 2012-05-05
Hi, I have a production PostgreSQL database used by a 24/7 application.  I have it running on RHEL 4.2 server running PostgreSQL 8.1.2 on SUN v20Z servers & thankfully running RH Cluster server (helpped me out today :)  This morning it decided to crash but was brought back online by the cluster software within a matter of seconds. The thing is I am left with the situation of trying to work out why it happend. It appeared to run out of memory (Out of Memory: Killed process 22431 (postmaster).
) so could be a Postgres bug. Does any one have idea's of where to start?  Below is the output during the incident logged in the /var/log/messages file

Thanks,
Sel

Apr 18 11:46:02 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:46:02 node1 su(pam_unix)[8178]: session opened for user postgres by (uid=0)
Apr 18 11:46:03 node1 su(pam_unix)[8178]: session closed for user postgres
Apr 18 11:48:24 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:24 node1 kernel: Mem-info:
Apr 18 11:48:24 node1 kernel: DMA per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:24 node1 kernel: Normal per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:24 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:24 node1 kernel:
Apr 18 11:48:24 node1 kernel: Free pages:       13004kB (0kB HighMem)
Apr 18 11:48:24 node1 kernel: Active:225429 inactive:223531 dirty:0 writeback:0 unstable:0 free:3251 slab:49026 mapped:446637 pagetables:6684
Apr 18 11:48:24 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:48:26 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:901716kB inactive:894124kB present:2080192kB pages_scanned:2154735 all_unreclaimable? yes
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:48:29 node1 kernel: Normal: 10*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB <7>Losing some ticks... checking if CPU frequency changed.
Apr 18 11:48:29 node1 kernel: = 1408kB
Apr 18 11:48:31 node1 kernel: HighMem: empty
Apr 18 11:48:34 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:48:34 node1 kernel: Free swap:            0kB
Apr 18 11:48:34 node1 kernel: 524144 pages of RAM
Apr 18 11:48:34 node1 kernel: 10384 reserved pages
Apr 18 11:48:34 node1 kernel: 4113 pages shared
Apr 18 11:48:34 node1 kernel: 0 pages swap cached
Apr 18 11:48:34 node1 kernel: Out of Memory: Killed process 22429 (postmaster).
Apr 18 11:48:34 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:34 node1 kernel: Mem-info:
Apr 18 11:48:36 node1 kernel: DMA per-cpu:
Apr 18 11:48:39 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:45 node1 kernel: Normal per-cpu:
Apr 18 11:48:45 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:45 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:46 node1 kernel:
Apr 18 11:48:49 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:48:52 node1 kernel: Active:192022 inactive:257223 dirty:0 writeback:0 unstable:0 free:3249 slab:48954 mapped:446565 pagetables:6658
Apr 18 11:49:10 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:50:00 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:02 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:768088kB inactive:1028892kB present:2080192kB pages_scanned:3202287 all_unreclaimable? yes
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:26 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:28 node1 kernel: HighMem: empty
Apr 18 11:51:29 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:30 node1 kernel: Free swap:            0kB
Apr 18 11:51:30 node1 kernel: 524144 pages of RAM
Apr 18 11:51:31 node1 kernel: 10384 reserved pages
Apr 18 11:51:31 node1 kernel: 3216 pages shared
Apr 18 11:51:31 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:51:31 node1 su[7499]: Warning!  Could not relabel /dev/pts/1 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:31 node1 su[9596]: Warning!  Could not relabel /dev/pts/3 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:31 node1 su[17331]: Warning!  Could not relabel /dev/pts/7 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:31 node1 su[13172]: Warning!  Could not relabel /dev/pts/4 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:32 node1 kernel: 0 pages swap cached
Apr 18 11:51:32 node1 su(pam_unix)[7499]: session closed for user root
Apr 18 11:51:32 node1 su(pam_unix)[9596]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[17331]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[13172]: session closed for user root
Apr 18 11:51:33 node1 kernel: Out of Memory: Killed process 22424 (postmaster).
Apr 18 11:51:33 node1 su(pam_unix)[8339]: session opened for user postgres by (uid=0)
Apr 18 11:51:33 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:34 node1 kernel: Mem-info:
Apr 18 11:51:34 node1 kernel: DMA per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:34 node1 kernel: Normal per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:34 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:34 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:234490 inactive:215587 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:937960kB inactive:862348kB present:2080192kB pages_scanned:5615412 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap:            0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22430 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:35 node1 kernel: Normal per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:35 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:35 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:256856 inactive:193221 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:1027424kB inactive:772884kB present:2080192kB pages_scanned:6180966 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap:            0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22431 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 su(pam_unix)[8339]: session closed for user postgres
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> status on script "Postgress App" returned 1 (generic error)
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:36 node1 kernel: Normal per-cpu:
Apr 18 11:51:36 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:36 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:36 node1 kernel:
Apr 18 11:51:36 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:51:36 node1 kernel: Active:224225 inactive:224642 dirty:0 writeback:0 unstable:0 free:3249 slab:48923 mapped:446679 pagetables:6589
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> Stopping service Postgres Database Cluster
Apr 18 11:51:36 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:896900kB inactive:898568kB present:2080192kB pages_scanned:8011971 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:36 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh stop
Apr 18 11:51:36 node1 kernel: Normal: 0*4kB 15*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session opened for user postgres by (uid=0)
Apr 18 11:51:36 node1 kernel: HighMem: empty
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session closed for user postgres
Apr 18 11:51:37 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:37 node1 kernel: Free swap:            0kB
Apr 18 11:51:37 node1 kernel: 524144 pages of RAM
Apr 18 11:51:37 node1 kernel: 10384 reserved pages
Apr 18 11:51:37 node1 kernel: 3220 pages shared
Apr 18 11:51:37 node1 kernel: 0 pages swap cached
Apr 18 11:51:37 node1 kernel: Out of Memory: Killed process 4092 (gnome-terminal).
Apr 18 11:51:37 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:37 node1 kernel: Mem-info:
Apr 18 11:51:37 node1 clurgmgrd: [15304]: <info> Removing IPv4 address 10.2.33.45 from eth0
Apr 18 11:51:37 node1 kernel: DMA per-cpu:
Apr 18 11:51:37 node1 gconfd (sels-8420): starting (version 2.8.1), pid 8420 user 'sels'
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:37 node1 kernel: Normal per-cpu:
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
Apr 18 11:51:37 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readwrite:/home/sels/.gconf" to a writable configuration source at position 1
Apr 18 11:51:38 node1 kernel:
Apr 18 11:51:38 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Apr 18 11:51:38 node1 kernel: Free pages:       13004kB (0kB HighMem)
Apr 18 11:51:38 node1 kernel: Active:225234 inactive:223793 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:38 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:900936kB inactive:895172kB present:2080192kB pages_scanned:3741771 all_unreclaimable? no
Apr 18 11:51:39 node1 sshd(pam_unix)[8479]: session opened for user root by (uid=0)
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436843, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap:            0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 3060 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 2927 (nautilus).
Apr 18 11:51:40 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:40 node1 kernel: Mem-info:
Apr 18 11:51:40 node1 kernel: DMA per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:40 node1 kernel: Normal per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:40 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:40 node1 kernel:
Apr 18 11:51:40 node1 kernel: Free pages:       13004kB (0kB HighMem)
Apr 18 11:51:40 node1 kernel: Active:239766 inactive:209805 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:40 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:959064kB inactive:839220kB present:2080192kB pages_scanned:7270296 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436847, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap:            0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 2516 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 3074 (nautilus).
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <notice> Forcefully unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22433 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22434 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22435 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:47 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster is recovering
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Recovering failed service Postgres Database Cluster
Apr 18 11:51:48 node1 kernel: kjournald starting.  Commit interval 5 seconds
Apr 18 11:51:48 node1 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Apr 18 11:51:48 node1 kernel: EXT3 FS on sdb1, internal journal
Apr 18 11:51:48 node1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Apr 18 11:51:48 node1 kernel: SELinux: initialized (dev sdb1, type ext3), uses xattr
Apr 18 11:51:49 node1 clurgmgrd: [15304]: <info> Adding IPv4 address 10.2.33.45 to eth0
Apr 18 11:51:50 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh start
Apr 18 11:51:50 node1 su(pam_unix)[8713]: session opened for user postgres by (uid=0)
Apr 18 11:51:51 node1 su(pam_unix)[8713]: session closed for user postgres
Apr 18 11:51:51 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster started
0
Comment
Question by:sel999
  • 3
  • 2
6 Comments
 
LVL 15

Expert Comment

by:m1tk4
ID: 16479895
I would not point fingers at Postgres right away - nautilus and gnome-terminal were killed as well for the same reasons. I would start with profiling overall memory usage and looking at what application actually ate up all your memory - it could be that it wasn't Postgres.

There is also a possibility that you are running too many child processes with Postgres and they run out of memory.
0
 
LVL 62

Accepted Solution

by:
gheist earned 225 total points
ID: 16483362
> Out of Memory: Killed process 3074 (nautilus).
When out of memory your system kills processes. This is normal.
After postmaster dies use your creativity with "ipcs -a" and "ipcrm -m" to clean up
Post output of "free" and "hdparm /dev/hda" and top lines of "dmesg | less" down to memory sizes.

0
 
LVL 62

Expert Comment

by:gheist
ID: 16490902
What is so incomplete in my request for additional system information ???
0
Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

 

Author Comment

by:sel999
ID: 16497912
Hiya, not sure what you mean?
0
 
LVL 62

Expert Comment

by:gheist
ID: 16502418
Why B grade ???
0
 

Author Comment

by:sel999
ID: 16505647
Oh I don't actaully remember setting a grade. I'm sorry I would have given a A grade but don't remember choosing a grade at all. I will see if I can change it. Rgs, Sel
0

Featured Post

Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Best database to use for Maps is PostgreSQL. This is an open source database. Comes as a package with most Linux OS. For more info visit the following site: http://www.postgresql.org/ (http://www.postgresql.org/) This requires some add-o…
By, Vadim Tkachenko. In this article we’ll look at ClickHouse on its one year anniversary.
Steps to create a PostgreSQL RDS instance in the Amazon cloud. We will cover some of the default settings and show how to connect to the instance once it is up and running.
Is your data getting by on basic protection measures? In today’s climate of debilitating malware and ransomware—like WannaCry—that may not be enough. You need to establish more than basics, like a recovery plan that protects both data and endpoints.…
Suggested Courses

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question