Solved

CentOS 5.6 box locking up with memory error

Posted on 2012-04-05
3
516 Views
Last Modified: 2012-06-21
I've just started getting this memory error and I'm not sure why. I'm utilizing quite a bit of squid processes and a "tunnel" process (proxy server), but this has never happened before. It's locking up the box and making it completely unusable.

I was able to view the console through xenserver. I have to force a reboot every time this happens.

I also have httpd/mysqld/vsftpd running but they are not doing very many requests. The problem started happening a few days after installing httpd/mysqld/vsftpd. Prior to this, the uptime on the box was over 6 months.

Linux proxyserver 2.6.18-308.1.1.el5xen #1 SMP Wed Mar 7 04:57:35 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff802c1bc6>] out_of_memory+0x8b/0x203
 [<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
 [<ffffffff80213a4f>] __do_page_cache_readahead+0x13a/0x1af
 [<ffffffff802c0990>] page_cache_read+0x51/0xba
 [<ffffffff8021430d>] filemap_nopage+0x20b/0x360
 [<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
 [<ffffffff8020d5b2>] do_sync_read+0xc7/0x104
 [<ffffffff8023459f>] lock_sock+0xa7/0xb2
 [<ffffffff80266d94>] do_page_fault+0xf72/0x131b
 [<ffffffff8029dc2e>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8025f82b>] error_exit+0x0/0x6e

DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
cpu 4 hot: high 0, batch 1 used:0
cpu 4 cold: high 0, batch 1 used:0
cpu 5 hot: high 0, batch 1 used:0
cpu 5 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:165
cpu 0 cold: high 62, batch 15 used:45
cpu 1 hot: high 186, batch 31 used:21
cpu 1 cold: high 62, batch 15 used:50
cpu 2 hot: high 186, batch 31 used:29
cpu 2 cold: high 62, batch 15 used:45
cpu 3 hot: high 186, batch 31 used:29
cpu 3 cold: high 62, batch 15 used:49
cpu 4 hot: high 186, batch 31 used:28
cpu 4 cold: high 62, batch 15 used:42
cpu 5 hot: high 186, batch 31 used:27
cpu 5 cold: high 62, batch 15 used:51
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:56
cpu 0 cold: high 62, batch 15 used:61
cpu 1 hot: high 186, batch 31 used:164
cpu 1 cold: high 62, batch 15 used:13
cpu 2 hot: high 186, batch 31 used:164
cpu 2 cold: high 62, batch 15 used:6
cpu 3 hot: high 186, batch 31 used:172
cpu 3 cold: high 62, batch 15 used:61
cpu 4 hot: high 186, batch 31 used:175
cpu 4 cold: high 62, batch 15 used:25
cpu 5 hot: high 186, batch 31 used:184
cpu 5 cold: high 62, batch 15 used:38
HighMem per-cpu: empty
Free pages:      403280kB (0kB HighMem)
Active:1179951 inactive:0 dirty:0 writeback:0 unstable:0 free:100820 slab:119873 mapped-file:980 mapped-anon:1180891 pagetables:6710
DMA free:3272kB min:1572kB low:1964kB high:2356kB active:0kB inactive:0kB present:9076kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 4024 5902 5902
DMA32 free:275168kB min:713864kB low:892328kB high:1070796kB active:3258028kB inactive:0kB present:4120800kB pages_scanned:5272893537 all_unreclaimable? yes
lowmem_reserve[]: 0 0 1877 1877
Normal free:124840kB min:333136kB low:416420kB high:499704kB active:1461576kB inactive:0kB present:1923040kB pages_scanned:1840806304 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 1*8kB 2*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 0*4096kB = 32

Open in new window


Any idea what this means?
0
Comment
Question by:mcainc
  • 2
3 Comments
 
LVL 11

Accepted Solution

by:
legolasthehansy earned 500 total points
ID: 37811116
You are victim of the OOM Killer (out of memory error)
It looks like your httpd process is eating off memory. Monitor your server extensively and try to check what causes it. Has anything changed recently? That should be a start.
0
 

Author Comment

by:mcainc
ID: 37813717
The only thing that changed was that I added httpd/mysqld/vsftpd and then this started happening a couple of days later (when I began sending traffic).

Right now I'm looking at the overview of the server (via XenCenter) and there's only 9% CPU utilization and 969 of 6000mb of memory in use. This is with a reasonable traffic load.

The confusing part is that the machine was receiving relatively low amounts of traffic around both times that it crashed (once at 2am and once at 4am).

I also noticed that this machine does not have any swap configured. Could that help?
0
 

Author Closing Comment

by:mcainc
ID: 37906840
thanks
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Can't see Ubuntu desktop on KVM switch unless I reboot the PC 22 83
SMB share across internet 15 64
expand ext4 on centos 6 5 37
trouble on installing syslog-ng on CentOS 7 7 52
This article will explain how to establish a SSH connection to Ubuntu through the firewall and using a different port other then 22. I have set up a Ubuntu virtual machine in Virtualbox and I am running a Windows 7 workstation. From the Ubuntu vi…
The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

791 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question