Solved

CentOS 5.6 box locking up with memory error

Posted on 2012-04-05
3
513 Views
Last Modified: 2012-06-21
I've just started getting this memory error and I'm not sure why. I'm utilizing quite a bit of squid processes and a "tunnel" process (proxy server), but this has never happened before. It's locking up the box and making it completely unusable.

I was able to view the console through xenserver. I have to force a reboot every time this happens.

I also have httpd/mysqld/vsftpd running but they are not doing very many requests. The problem started happening a few days after installing httpd/mysqld/vsftpd. Prior to this, the uptime on the box was over 6 months.

Linux proxyserver 2.6.18-308.1.1.el5xen #1 SMP Wed Mar 7 04:57:35 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff802c1bc6>] out_of_memory+0x8b/0x203
 [<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
 [<ffffffff80213a4f>] __do_page_cache_readahead+0x13a/0x1af
 [<ffffffff802c0990>] page_cache_read+0x51/0xba
 [<ffffffff8021430d>] filemap_nopage+0x20b/0x360
 [<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
 [<ffffffff8020d5b2>] do_sync_read+0xc7/0x104
 [<ffffffff8023459f>] lock_sock+0xa7/0xb2
 [<ffffffff80266d94>] do_page_fault+0xf72/0x131b
 [<ffffffff8029dc2e>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8025f82b>] error_exit+0x0/0x6e

DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
cpu 4 hot: high 0, batch 1 used:0
cpu 4 cold: high 0, batch 1 used:0
cpu 5 hot: high 0, batch 1 used:0
cpu 5 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:165
cpu 0 cold: high 62, batch 15 used:45
cpu 1 hot: high 186, batch 31 used:21
cpu 1 cold: high 62, batch 15 used:50
cpu 2 hot: high 186, batch 31 used:29
cpu 2 cold: high 62, batch 15 used:45
cpu 3 hot: high 186, batch 31 used:29
cpu 3 cold: high 62, batch 15 used:49
cpu 4 hot: high 186, batch 31 used:28
cpu 4 cold: high 62, batch 15 used:42
cpu 5 hot: high 186, batch 31 used:27
cpu 5 cold: high 62, batch 15 used:51
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:56
cpu 0 cold: high 62, batch 15 used:61
cpu 1 hot: high 186, batch 31 used:164
cpu 1 cold: high 62, batch 15 used:13
cpu 2 hot: high 186, batch 31 used:164
cpu 2 cold: high 62, batch 15 used:6
cpu 3 hot: high 186, batch 31 used:172
cpu 3 cold: high 62, batch 15 used:61
cpu 4 hot: high 186, batch 31 used:175
cpu 4 cold: high 62, batch 15 used:25
cpu 5 hot: high 186, batch 31 used:184
cpu 5 cold: high 62, batch 15 used:38
HighMem per-cpu: empty
Free pages:      403280kB (0kB HighMem)
Active:1179951 inactive:0 dirty:0 writeback:0 unstable:0 free:100820 slab:119873 mapped-file:980 mapped-anon:1180891 pagetables:6710
DMA free:3272kB min:1572kB low:1964kB high:2356kB active:0kB inactive:0kB present:9076kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 4024 5902 5902
DMA32 free:275168kB min:713864kB low:892328kB high:1070796kB active:3258028kB inactive:0kB present:4120800kB pages_scanned:5272893537 all_unreclaimable? yes
lowmem_reserve[]: 0 0 1877 1877
Normal free:124840kB min:333136kB low:416420kB high:499704kB active:1461576kB inactive:0kB present:1923040kB pages_scanned:1840806304 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 1*8kB 2*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 0*4096kB = 32

Open in new window


Any idea what this means?
0
Comment
Question by:mcainc
  • 2
3 Comments
 
LVL 11

Accepted Solution

by:
legolasthehansy earned 500 total points
Comment Utility
You are victim of the OOM Killer (out of memory error)
It looks like your httpd process is eating off memory. Monitor your server extensively and try to check what causes it. Has anything changed recently? That should be a start.
0
 

Author Comment

by:mcainc
Comment Utility
The only thing that changed was that I added httpd/mysqld/vsftpd and then this started happening a couple of days later (when I began sending traffic).

Right now I'm looking at the overview of the server (via XenCenter) and there's only 9% CPU utilization and 969 of 6000mb of memory in use. This is with a reasonable traffic load.

The confusing part is that the machine was receiving relatively low amounts of traffic around both times that it crashed (once at 2am and once at 4am).

I also noticed that this machine does not have any swap configured. Could that help?
0
 

Author Closing Comment

by:mcainc
Comment Utility
thanks
0

Featured Post

Free Gift Card with Acronis Backup Purchase!

Backup any data in any location: local and remote systems, physical and virtual servers, private and public clouds, Macs and PCs, tablets and mobile devices, & more! For limited time only, buy any Acronis backup products and get a FREE Amazon/Best Buy gift card worth up to $200!

Join & Write a Comment

If you use Debian 6 Squeeze and you are tired of looking at the childish graphical GDM login screen that is used by default, here's an easy way to change it. If you've already tried to change it you've probably discovered that none of the old met…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now