Link to home
Start Free TrialLog in
Avatar of sel999
sel999

asked on

Postgres 8.1.2 crashed now how to figure out why?

Hi, I have a production PostgreSQL database used by a 24/7 application.  I have it running on RHEL 4.2 server running PostgreSQL 8.1.2 on SUN v20Z servers & thankfully running RH Cluster server (helpped me out today :)  This morning it decided to crash but was brought back online by the cluster software within a matter of seconds. The thing is I am left with the situation of trying to work out why it happend. It appeared to run out of memory (Out of Memory: Killed process 22431 (postmaster).
) so could be a Postgres bug. Does any one have idea's of where to start?  Below is the output during the incident logged in the /var/log/messages file

Thanks,
Sel

Apr 18 11:46:02 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:46:02 node1 su(pam_unix)[8178]: session opened for user postgres by (uid=0)
Apr 18 11:46:03 node1 su(pam_unix)[8178]: session closed for user postgres
Apr 18 11:48:24 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:24 node1 kernel: Mem-info:
Apr 18 11:48:24 node1 kernel: DMA per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:24 node1 kernel: Normal per-cpu:
Apr 18 11:48:24 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:24 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:24 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:24 node1 kernel:
Apr 18 11:48:24 node1 kernel: Free pages:       13004kB (0kB HighMem)
Apr 18 11:48:24 node1 kernel: Active:225429 inactive:223531 dirty:0 writeback:0 unstable:0 free:3251 slab:49026 mapped:446637 pagetables:6684
Apr 18 11:48:24 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:48:26 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:901716kB inactive:894124kB present:2080192kB pages_scanned:2154735 all_unreclaimable? yes
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:48:29 node1 kernel: protections[]: 0 0 0
Apr 18 11:48:29 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:48:29 node1 kernel: Normal: 10*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB <7>Losing some ticks... checking if CPU frequency changed.
Apr 18 11:48:29 node1 kernel: = 1408kB
Apr 18 11:48:31 node1 kernel: HighMem: empty
Apr 18 11:48:34 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:48:34 node1 kernel: Free swap:            0kB
Apr 18 11:48:34 node1 kernel: 524144 pages of RAM
Apr 18 11:48:34 node1 kernel: 10384 reserved pages
Apr 18 11:48:34 node1 kernel: 4113 pages shared
Apr 18 11:48:34 node1 kernel: 0 pages swap cached
Apr 18 11:48:34 node1 kernel: Out of Memory: Killed process 22429 (postmaster).
Apr 18 11:48:34 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:48:34 node1 kernel: Mem-info:
Apr 18 11:48:36 node1 kernel: DMA per-cpu:
Apr 18 11:48:39 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:48:45 node1 kernel: Normal per-cpu:
Apr 18 11:48:45 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:48:45 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:48:45 node1 kernel: HighMem per-cpu: empty
Apr 18 11:48:46 node1 kernel:
Apr 18 11:48:49 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:48:52 node1 kernel: Active:192022 inactive:257223 dirty:0 writeback:0 unstable:0 free:3249 slab:48954 mapped:446565 pagetables:6658
Apr 18 11:49:10 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:50:00 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:02 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:768088kB inactive:1028892kB present:2080192kB pages_scanned:3202287 all_unreclaimable? yes
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:50:14 node1 kernel: protections[]: 0 0 0
Apr 18 11:50:14 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:26 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:28 node1 kernel: HighMem: empty
Apr 18 11:51:29 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:30 node1 kernel: Free swap:            0kB
Apr 18 11:51:30 node1 kernel: 524144 pages of RAM
Apr 18 11:51:31 node1 kernel: 10384 reserved pages
Apr 18 11:51:31 node1 kernel: 3216 pages shared
Apr 18 11:51:31 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh status
Apr 18 11:51:31 node1 su[7499]: Warning!  Could not relabel /dev/pts/1 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:31 node1 su[9596]: Warning!  Could not relabel /dev/pts/3 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:31 node1 su[17331]: Warning!  Could not relabel /dev/pts/7 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:31 node1 su[13172]: Warning!  Could not relabel /dev/pts/4 with user_u:object_r:initrc_devpts_t, not relabeling.
Apr 18 11:51:32 node1 kernel: 0 pages swap cached
Apr 18 11:51:32 node1 su(pam_unix)[7499]: session closed for user root
Apr 18 11:51:32 node1 su(pam_unix)[9596]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[17331]: session closed for user root
Apr 18 11:51:33 node1 su(pam_unix)[13172]: session closed for user root
Apr 18 11:51:33 node1 kernel: Out of Memory: Killed process 22424 (postmaster).
Apr 18 11:51:33 node1 su(pam_unix)[8339]: session opened for user postgres by (uid=0)
Apr 18 11:51:33 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:34 node1 kernel: Mem-info:
Apr 18 11:51:34 node1 kernel: DMA per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:34 node1 kernel: Normal per-cpu:
Apr 18 11:51:34 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:34 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:34 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:34 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:234490 inactive:215587 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:937960kB inactive:862348kB present:2080192kB pages_scanned:5615412 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap:            0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22430 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:35 node1 kernel: Normal per-cpu:
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:35 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:35 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:35 node1 kernel:
Apr 18 11:51:35 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:51:35 node1 kernel: Active:256856 inactive:193221 dirty:0 writeback:0 unstable:0 free:3249 slab:48952 mapped:446565 pagetables:6658
Apr 18 11:51:35 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502169 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:1027424kB inactive:772884kB present:2080192kB pages_scanned:6180966 all_unreclaimable? yes
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:35 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:35 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:35 node1 kernel: Normal: 8*4kB 13*8kB 7*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:35 node1 kernel: HighMem: empty
Apr 18 11:51:35 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:35 node1 kernel: Free swap:            0kB
Apr 18 11:51:35 node1 kernel: 524144 pages of RAM
Apr 18 11:51:35 node1 kernel: 10384 reserved pages
Apr 18 11:51:35 node1 kernel: 2384 pages shared
Apr 18 11:51:35 node1 kernel: 0 pages swap cached
Apr 18 11:51:35 node1 kernel: Out of Memory: Killed process 22431 (postmaster).
Apr 18 11:51:35 node1 kernel: oom-killer: gfp_mask=0xd0
Apr 18 11:51:35 node1 kernel: Mem-info:
Apr 18 11:51:35 node1 kernel: DMA per-cpu:
Apr 18 11:51:35 node1 su(pam_unix)[8339]: session closed for user postgres
Apr 18 11:51:35 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> status on script "Postgress App" returned 1 (generic error)
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:36 node1 kernel: Normal per-cpu:
Apr 18 11:51:36 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:36 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:36 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:36 node1 kernel:
Apr 18 11:51:36 node1 kernel: Free pages:       12996kB (0kB HighMem)
Apr 18 11:51:36 node1 kernel: Active:224225 inactive:224642 dirty:0 writeback:0 unstable:0 free:3249 slab:48923 mapped:446679 pagetables:6589
Apr 18 11:51:36 node1 clurgmgrd[15304]: <notice> Stopping service Postgres Database Cluster
Apr 18 11:51:36 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: Normal free:1400kB min:1428kB low:2856kB high:4284kB active:896900kB inactive:898568kB present:2080192kB pages_scanned:8011971 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:36 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:36 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:36 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh stop
Apr 18 11:51:36 node1 kernel: Normal: 0*4kB 15*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1400kB
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session opened for user postgres by (uid=0)
Apr 18 11:51:36 node1 kernel: HighMem: empty
Apr 18 11:51:36 node1 su(pam_unix)[8376]: session closed for user postgres
Apr 18 11:51:37 node1 kernel: Swap cache: add 586816, delete 586816, find 1426745/1436842, race 0+0
Apr 18 11:51:37 node1 kernel: Free swap:            0kB
Apr 18 11:51:37 node1 kernel: 524144 pages of RAM
Apr 18 11:51:37 node1 kernel: 10384 reserved pages
Apr 18 11:51:37 node1 kernel: 3220 pages shared
Apr 18 11:51:37 node1 kernel: 0 pages swap cached
Apr 18 11:51:37 node1 kernel: Out of Memory: Killed process 4092 (gnome-terminal).
Apr 18 11:51:37 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:37 node1 kernel: Mem-info:
Apr 18 11:51:37 node1 clurgmgrd: [15304]: <info> Removing IPv4 address 10.2.33.45 from eth0
Apr 18 11:51:37 node1 kernel: DMA per-cpu:
Apr 18 11:51:37 node1 gconfd (sels-8420): starting (version 2.8.1), pid 8420 user 'sels'
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:37 node1 kernel: Normal per-cpu:
Apr 18 11:51:37 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:37 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
Apr 18 11:51:37 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:37 node1 gconfd (sels-8420): Resolved address "xml:readwrite:/home/sels/.gconf" to a writable configuration source at position 1
Apr 18 11:51:38 node1 kernel:
Apr 18 11:51:38 node1 gconfd (sels-8420): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Apr 18 11:51:38 node1 kernel: Free pages:       13004kB (0kB HighMem)
Apr 18 11:51:38 node1 kernel: Active:225234 inactive:223793 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:38 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:900936kB inactive:895172kB present:2080192kB pages_scanned:3741771 all_unreclaimable? no
Apr 18 11:51:39 node1 sshd(pam_unix)[8479]: session opened for user root by (uid=0)
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:39 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:39 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436843, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap:            0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 3060 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 2927 (nautilus).
Apr 18 11:51:40 node1 kernel: oom-killer: gfp_mask=0x1d2
Apr 18 11:51:40 node1 kernel: Mem-info:
Apr 18 11:51:40 node1 kernel: DMA per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 18 11:51:40 node1 kernel: Normal per-cpu:
Apr 18 11:51:40 node1 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 18 11:51:40 node1 kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 18 11:51:40 node1 kernel: HighMem per-cpu: empty
Apr 18 11:51:40 node1 kernel:
Apr 18 11:51:40 node1 kernel: Free pages:       13004kB (0kB HighMem)
Apr 18 11:51:40 node1 kernel: Active:239766 inactive:209805 dirty:0 writeback:0 unstable:0 free:3251 slab:48916 mapped:446679 pagetables:6589
Apr 18 11:51:40 node1 kernel: DMA free:11596kB min:8kB low:16kB high:24kB active:0kB inactive:0kB present:16384kB pages_scanned:502170 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: Normal free:1408kB min:1428kB low:2856kB high:4284kB active:959064kB inactive:839220kB present:2080192kB pages_scanned:7270296 all_unreclaimable? yes
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Apr 18 11:51:40 node1 kernel: protections[]: 0 0 0
Apr 18 11:51:40 node1 kernel: DMA: 3*4kB 4*8kB 4*16kB 3*32kB 4*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11596kB
Apr 18 11:51:40 node1 kernel: Normal: 0*4kB 16*8kB 8*16kB 4*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1408kB
Apr 18 11:51:40 node1 kernel: HighMem: empty
Apr 18 11:51:40 node1 kernel: Swap cache: add 586825, delete 586825, find 1426745/1436847, race 0+0
Apr 18 11:51:40 node1 kernel: Free swap:            0kB
Apr 18 11:51:40 node1 kernel: 524144 pages of RAM
Apr 18 11:51:40 node1 kernel: 10384 reserved pages
Apr 18 11:51:40 node1 kernel: 2516 pages shared
Apr 18 11:51:40 node1 kernel: 0 pages swap cached
Apr 18 11:51:40 node1 kernel: Out of Memory: Killed process 3074 (nautilus).
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <notice> Forcefully unmounting /usr/local/pgsql/data
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22433 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22434 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:41 node1 clurgmgrd: [15304]: <warning> killing process 22435 (postgres postmaste /usr/local/pgsql/data)
Apr 18 11:51:47 node1 clurgmgrd: [15304]: <info> unmounting /usr/local/pgsql/data
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster is recovering
Apr 18 11:51:48 node1 clurgmgrd[15304]: <notice> Recovering failed service Postgres Database Cluster
Apr 18 11:51:48 node1 kernel: kjournald starting.  Commit interval 5 seconds
Apr 18 11:51:48 node1 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Apr 18 11:51:48 node1 kernel: EXT3 FS on sdb1, internal journal
Apr 18 11:51:48 node1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Apr 18 11:51:48 node1 kernel: SELinux: initialized (dev sdb1, type ext3), uses xattr
Apr 18 11:51:49 node1 clurgmgrd: [15304]: <info> Adding IPv4 address 10.2.33.45 to eth0
Apr 18 11:51:50 node1 clurgmgrd: [15304]: <info> Executing /etc/init.d/postgres.sh start
Apr 18 11:51:50 node1 su(pam_unix)[8713]: session opened for user postgres by (uid=0)
Apr 18 11:51:51 node1 su(pam_unix)[8713]: session closed for user postgres
Apr 18 11:51:51 node1 clurgmgrd[15304]: <notice> Service Postgres Database Cluster started
Avatar of m1tk4
m1tk4
Flag of United States of America image

I would not point fingers at Postgres right away - nautilus and gnome-terminal were killed as well for the same reasons. I would start with profiling overall memory usage and looking at what application actually ate up all your memory - it could be that it wasn't Postgres.

There is also a possibility that you are running too many child processes with Postgres and they run out of memory.
ASKER CERTIFIED SOLUTION
Avatar of gheist
gheist
Flag of Belgium image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
What is so incomplete in my request for additional system information ???
Avatar of sel999
sel999

ASKER

Hiya, not sure what you mean?
Why B grade ???
Avatar of sel999

ASKER

Oh I don't actaully remember setting a grade. I'm sorry I would have given a A grade but don't remember choosing a grade at all. I will see if I can change it. Rgs, Sel