We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you a podcast all about Citrix Workspace, moving to the cloud, and analytics & intelligence. Episode 2 coming soon!Listen Now

x

How to check which process caused High CPU in Linux?

Medium Priority
168 Views
Last Modified: 2019-11-10
We got some threshold alert on CPU usage sometimes. Thus we want to find out which process caused high CPU utilization on Linux.
After we use top and ntop command, all processes used only 0.7 to 1% CPU resources from time to time. We have no idea which process caused the issue.
Any command or tools could help us with this problem? Thank you.

RedHat and Ubuntu.
Comment
Watch Question

Fractional CTO
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
Likely best to use the SAR - System Activity Report tool for collecting this type of information.

CPU usage occurs in realtime, so to determine high CPU users at a specific point in time requires historical data (via SAR or some other tool), allowing you to look back in time to determine high CPU usage.

This means you can't make a determination (unless SAR was running) of any past CPU usage.

To capture this data for future CPU usage spikes, install SAR now... which will begin collecting historical data...

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts
David FavorFractional CTO
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
CPU tracking tricks...

1) You can run this command...

echo "#####" && date && top -i -b -d 1 > /var/log/top.log 2>&1

Open in new window


This will capture a running log of active processes every 1 second. Sometimes this is highly useful to quickly make determinations about CPU hogs.

2) You can run this command...

csysdig -vcontainers

Open in new window


This will break down CPU usage by containers, if you're running many LXD or Docker or other containers on your machine.
David FavorFractional CTO
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
Consideration: High CPU usage almost always related to high I/O, usually disk writes, as disk reads should primarily come from accessing memory buffers.

Another trick is to run this command...

iotop -P -a -d 1

Open in new window


Then high left arrow till Disk Writes becomes the sort column.

Almost always, the worst disk hog is also the worst CPU hog.

For example, here's the output of the above command off a random server...

# echo "#####" && date && iotop -o -b -n 1 -d 1
#####
Sun Oct 20 15:16:32 UTC 2019
Total DISK READ :      10.36 M/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 205302 be/4 1000107     5.88 M/s    0.00 B/s  0.00 %  4.62 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/mariadb-error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
 198236 be/4 1000107     2.80 M/s    0.00 B/s  0.00 %  2.12 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/mariadb-error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
 208784 be/4 1000107  1720.54 K/s    0.00 B/s  0.00 %  1.33 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/mariadb-error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

Open in new window


You can also enable disk i/o debugging, via...

echo 1 > /proc/sys/vm/block_dump

Open in new window


Which will (usually) dump statistics into /var/log/syslog or /var/log/messages or some other log file.
nociSoftware Engineer
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
You can try to check the process tree: ps -axf  that might show a process which has a lot of  short running subprocesses. (f.e. nagios running its sentinels)
the process tree will then show the combination.
Prabhin MPDevOps Engineer
CERTIFIED EXPERT
Distinguished Expert 2018

Commented:
ps aux --sort=-%mem | awk 'NR<=10{print $0}'

try this which will give you the top memory utilized process
Prabhin MPDevOps Engineer
CERTIFIED EXPERT
Distinguished Expert 2018
Commented:
f you must use ps, try this:

ps aux | sort -nrk 3,3 | head -n 5
If you want something that's truly 'top'esq with constant updates, use watch

watch "ps aux | sort -nrk 3,3 | head -n 5"
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.