Solved

CentOS 5.6 box locking up with memory error

Posted on 2012-04-05
3
515 Views
Last Modified: 2012-06-21
I've just started getting this memory error and I'm not sure why. I'm utilizing quite a bit of squid processes and a "tunnel" process (proxy server), but this has never happened before. It's locking up the box and making it completely unusable.

I was able to view the console through xenserver. I have to force a reboot every time this happens.

I also have httpd/mysqld/vsftpd running but they are not doing very many requests. The problem started happening a few days after installing httpd/mysqld/vsftpd. Prior to this, the uptime on the box was over 6 months.

Linux proxyserver 2.6.18-308.1.1.el5xen #1 SMP Wed Mar 7 04:57:35 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff802c1bc6>] out_of_memory+0x8b/0x203
 [<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
 [<ffffffff80213a4f>] __do_page_cache_readahead+0x13a/0x1af
 [<ffffffff802c0990>] page_cache_read+0x51/0xba
 [<ffffffff8021430d>] filemap_nopage+0x20b/0x360
 [<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
 [<ffffffff8020d5b2>] do_sync_read+0xc7/0x104
 [<ffffffff8023459f>] lock_sock+0xa7/0xb2
 [<ffffffff80266d94>] do_page_fault+0xf72/0x131b
 [<ffffffff8029dc2e>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8025f82b>] error_exit+0x0/0x6e

DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
cpu 4 hot: high 0, batch 1 used:0
cpu 4 cold: high 0, batch 1 used:0
cpu 5 hot: high 0, batch 1 used:0
cpu 5 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:165
cpu 0 cold: high 62, batch 15 used:45
cpu 1 hot: high 186, batch 31 used:21
cpu 1 cold: high 62, batch 15 used:50
cpu 2 hot: high 186, batch 31 used:29
cpu 2 cold: high 62, batch 15 used:45
cpu 3 hot: high 186, batch 31 used:29
cpu 3 cold: high 62, batch 15 used:49
cpu 4 hot: high 186, batch 31 used:28
cpu 4 cold: high 62, batch 15 used:42
cpu 5 hot: high 186, batch 31 used:27
cpu 5 cold: high 62, batch 15 used:51
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:56
cpu 0 cold: high 62, batch 15 used:61
cpu 1 hot: high 186, batch 31 used:164
cpu 1 cold: high 62, batch 15 used:13
cpu 2 hot: high 186, batch 31 used:164
cpu 2 cold: high 62, batch 15 used:6
cpu 3 hot: high 186, batch 31 used:172
cpu 3 cold: high 62, batch 15 used:61
cpu 4 hot: high 186, batch 31 used:175
cpu 4 cold: high 62, batch 15 used:25
cpu 5 hot: high 186, batch 31 used:184
cpu 5 cold: high 62, batch 15 used:38
HighMem per-cpu: empty
Free pages:      403280kB (0kB HighMem)
Active:1179951 inactive:0 dirty:0 writeback:0 unstable:0 free:100820 slab:119873 mapped-file:980 mapped-anon:1180891 pagetables:6710
DMA free:3272kB min:1572kB low:1964kB high:2356kB active:0kB inactive:0kB present:9076kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 4024 5902 5902
DMA32 free:275168kB min:713864kB low:892328kB high:1070796kB active:3258028kB inactive:0kB present:4120800kB pages_scanned:5272893537 all_unreclaimable? yes
lowmem_reserve[]: 0 0 1877 1877
Normal free:124840kB min:333136kB low:416420kB high:499704kB active:1461576kB inactive:0kB present:1923040kB pages_scanned:1840806304 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 1*8kB 2*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 0*4096kB = 32

Open in new window


Any idea what this means?
0
Comment
Question by:mcainc
  • 2
3 Comments
 
LVL 11

Accepted Solution

by:
legolasthehansy earned 500 total points
ID: 37811116
You are victim of the OOM Killer (out of memory error)
It looks like your httpd process is eating off memory. Monitor your server extensively and try to check what causes it. Has anything changed recently? That should be a start.
0
 

Author Comment

by:mcainc
ID: 37813717
The only thing that changed was that I added httpd/mysqld/vsftpd and then this started happening a couple of days later (when I began sending traffic).

Right now I'm looking at the overview of the server (via XenCenter) and there's only 9% CPU utilization and 969 of 6000mb of memory in use. This is with a reasonable traffic load.

The confusing part is that the machine was receiving relatively low amounts of traffic around both times that it crashed (once at 2am and once at 4am).

I also noticed that this machine does not have any swap configured. Could that help?
0
 

Author Closing Comment

by:mcainc
ID: 37906840
thanks
0

Featured Post

Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
awk file 6 47
Linux script delete files 3 29
bash file 10 34
linux SFTP 8 37
This article will explain how to establish a SSH connection to Ubuntu through the firewall and using a different port other then 22. I have set up a Ubuntu virtual machine in Virtualbox and I am running a Windows 7 workstation. From the Ubuntu vi…
Fine Tune your automatic Updates for Ubuntu / Debian
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

815 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now