[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1664
  • Last Modified:

kswapd process is eating up the whole CPU in RHEL5. Anyone has any idea ?

kswapd process is eating up the whole CPU in RHEL5. Anyone has any idea ?

TY/SA
0
Sandy
Asked:
Sandy
  • 9
  • 7
  • 5
  • +2
4 Solutions
 
Joseph GanSystem AdminCommented:
Place "top" command output here.
0
 
SandyAuthor Commented:
0
 
Joseph GanSystem AdminCommented:
Dose this machine is running OmniVision with single CPU?
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 
SandyAuthor Commented:
yes it is.. Is omnivision anyway related to kswapd ?

TY/SA
0
 
Joseph GanSystem AdminCommented:
Any other applications, or only omnivision ?
0
 
SandyAuthor Commented:
it has some customers application, multiple java processes are also there which are eating up 20% of the CPU, Information is little sensitive hence can't post over here.

I am concerned about remaining part of CPU which is being consumed by attached  2 top processes..

TY/SA
0
 
Joseph GanSystem AdminCommented:
Understood. Any image processes are using more system resorces, plus 20% of the CPU for Java.
If it is single CPU machine, consider add more CPU and RAM.
0
 
SandyAuthor Commented:
i will definitely gonna send this recommendation to my customer but the point here is why this "kswapd" is consuming such high CPU ?
0
 
Joseph GanSystem AdminCommented:
"kswapd" is swap space management in response to memory demands greater than physically available for all processes.
So RAM shortage is part of the problem in this case.
0
 
gheistCommented:
kswapd is memory management.
run
# vmstat 5 5
and post the result
also uname -a

Any sort of virtualisation involved? 32bit or 64bit system?
0
 
SandyAuthor Commented:
Please find the output

[root@linux ~]# vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 4259388 346172 10481028    0    0    12    10    1    3  1  0 99  0  0
 0  0      0 4259512 346176 10481024    0    0     0    13 1008  144  0  0 100  0  0
 0  0      0 4259264 346176 10481028    0    0     0     7 1004  161  0  0 100  0  0
 0  0      0 4259264 346184 10481028    0    0     0    65 1010  153  0  0 100  0  0
 0  0      0 4259264 346200 10481020    0    0     0    17 1004  149  0  0 100  0  0
[root@linux ~]# uname -a
Linux linux 2.6.18-194.11.4.el5 #1 SMP Fri Sep 17 04:57:05 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@linux ~]# 

Open in new window


Yes, it is VM on VMware 64bit virtualization.

TY/SA
0
 
SandyAuthor Commented:
@ganjos : M sorry, i just checked it again i can confirm it using 3 CPU's.

below is the top header output

Tasks: 228 total,   1 running, 227 sleeping,   0 stopped,   0 zombie
Cpu0  :  1.0%us,  1.0%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16436904k total, 12178780k used,  4258124k free,   346528k buffers
Swap:        0k total,        0k used,        0k free, 10481036k cached

Open in new window

0
 
nociSoftware EngineerCommented:
According to the first screenshot you are out of memory.  The swapper most probably is evicting pages from memory so they can be replaced by pages from processes the get active.... In this case another active process that wants it pages back again etc. etc.

This indicates your machine has a severe lack of memory. Create a VM with more memory and you'll see the kswapd usage drop. If you cannot extend the memory you need a swapfile to juggle writable memory (which is hopefully not used ) to a file. It may also make performance go even worse.
The best option is to assign more memory.
0
 
gheistCommented:
You need to enable swap space right away. You run short on RAM.
0
 
SandyAuthor Commented:
OK. i will keep this recommendation ahead and will keep you posted for the same.

Thanks for you suggestions.

TY/SA
0
 
gheistCommented:
You cannot hotplug RAM in RHEL5, so only chance at this moment is to add swap space.
0
 
nociSoftware EngineerCommented:
@gheist, that entirely depends on if the system can be paused/stopped or not.
Your assumption is that cannot be done.
0
 
gheistCommented:
BTW given kernel version your RHEL is UNSUPPORTED.

Do you have RedHat subscription? If not please delete RedHat code the way you prefer.
0
 
tfewsterCommented:
The system is not swapping/paging (As shown by vmstat) and is not out of memory (both top and vmstat show > 50% of memory is not being used for real tasks, so is being used as disk cache, which should be freed if a process needs memory).

This is a well known bug in the implementation of kswapd in Red Hat (in particular), and has been seen in kernel versions from 2.1 through to 6.3. Do a web search for "kswapd high CPU".

Pointless tasks that seem to make the problem go away for a while are: 1) Rebooting  2) Updating your kernel  3) Adding more memory  4) Adding a swap partition that will never get used (Actually, some processes such as databases like to try to reserve a swap area as they start up, in case they need to be swapped out)

It seems to me that kswapd doesn't recognise that cached memory is actually free, and so goes into panic mode. And the caching process can be quite aggressive in grabbing free memory.

To free pagecache:    echo 1 > /proc/sys/vm/drop_caches
This should be safe to do on a running system, and kswapd usage should then go back to normal

Longer term, you could restrict caching: (Though unused memory is wasted memory!)
To check/set pagecache usage: # cat /proc/sys/vm/pagecache
100              
# Output depending on your release/kernel version! This is the default for RHEL5

To set pagecache policy temporarily: # echo "90" > /proc/sys/vm/pagecache

To make it permanent: # echo "vm.pagecache 90" >> /etc/sysctl.conf

It's not clear to me if that is 90% of all memory or just the free memory; You may also want to try adjusting the minimum left free in /proc/sys/vm/min_free_kbytes; It can been seen and manipulated temporarily in the same way; To make a permanent change: # echo vm.min_free_kbytes=1024 >> /etc/sysctl.conf

P.S. There's an interesting summary of the issue at http://www-01.ibm.com/support/docview.wss?uid=swg21665911 (quite recent but, as usual, no real solution)
0
 
gheistCommented:
Wonderful discovery.... Havent you noticed system has no swap file at all?
0
 
tfewsterCommented:
Hi gheist,

Yes, running without any swap space is a valid configuration and is becoming more common in these days of large-memory systems. And as the first `top` output shows 7.7 GB of memory being used for cache (i.e. available if needed), the system is NOT out of memory and therefore does NOT need to page/swap. (The second `top` output and `vmstat` show low CPU use and free memory, so are not relevant to this problem)

There are still valid reasons for having swap space; One example (from the Linux Kernel Mailing List FAQ, http://www.tux.org/lkml/#s9-14 ) is to allow little-used programs such as lpd to be moved out to swap, freeing up valuable RAM for caching :-)  

Regards,
Tim

P.S. I apologise if my first post sounded like a dig at other Experts, but the earlier comments will not fix the root cause of the problem
0
 
gheistCommented:
The root cause is that system has not been updated for last 5 years and all system software is that long out of support.

You can free cache memory by issuing:
sync ; sync
sysctl vm.drop_caches=3
sysctl vm.drop_caches=0
0
 
SandyAuthor Commented:
as per above valuable suggestions i am going fwd with below.


1. Dropping the cache.
2. Enabling least swap space
3. putting some monitoring mechanism to see on what instances consumption moves so fast.

TY/SA
0
 
gheistCommented:
0. Upgrade to supported release....
3. Is effectively handled by having swap space - if it does IO to swap file you need more RAM
0
 
SandyAuthor Commented:
Thank you everybody for your valuable suggestions. Thanks a lot..
0

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

  • 9
  • 7
  • 5
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now