High CPU Load
HI
I am running Xen-PV with SolusVM on Centos 6 and hosting about 11 Virtual machines. 5. Since yesterday the cpu load on my host machine goes very high and reaches between 28-35. Connecting with SSH becomes very hard. I run top command but did not find any processes taking very high CPU.
I am puzzled. I also ran xm top command (screenshot attached).
Also /var/log/messages show below message:
and
I would be really thankful if someone can please help me identify what is causing this high CPU load.
Screen-Shot-2015-10-19-at-8.42.17-PM.png
I am running Xen-PV with SolusVM on Centos 6 and hosting about 11 Virtual machines. 5. Since yesterday the cpu load on my host machine goes very high and reaches between 28-35. Connecting with SSH becomes very hard. I run top command but did not find any processes taking very high CPU.
I am puzzled. I also ran xm top command (screenshot attached).
Also /var/log/messages show below message:
Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: audit_lost=150 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: audit_lost=151 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: audit_lost=152 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit_log_start: 2 callbacks suppressed
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: audit_lost=154 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: audit_lost=155 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: audit_lost=156 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
and
# aureport --start today --event --summary -i
Event Summary Report
======================
total type
======================
6751 NETFILTER_CFG
1470 USER_START
1462 LOGIN
1462 USER_ACCT
1462 CRED_ACQ
1446 USER_END
1445 CRED_DISP
1415 CRYPTO_KEY_USER
748 USER_AUTH
584 CRYPTO_SESSION
538 USER_LOGIN
265 USER_ERR
11 ANOM_PROMISCUOUS
8 CRED_REFR
2 USER_LOGOUT
1 CONFIG_CHANGE
I would be really thankful if someone can please help me identify what is causing this high CPU load.
Screen-Shot-2015-10-19-at-8.42.17-PM.png
Do more with
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
if you have a lot of short lived processes, they may not show up in top, while causing a boatload of CPU usage.
Try to find out if there are some runaway server restarts...
The problem causing processes might also cause a lot of logging, that may help. Look for processes with a lot of fast changing pid's
Try to find out if there are some runaway server restarts...
The problem causing processes might also cause a lot of logging, that may help. Look for processes with a lot of fast changing pid's
Thanks for the reply.
> The problem causing processes might also cause a lot of logging, that may help. Look for processes with a lot of fast changing pid's
How can I see those processes when top doesn't show that? Sorry if the the question looks noob.
> The problem causing processes might also cause a lot of logging, that may help. Look for processes with a lot of fast changing pid's
How can I see those processes when top doesn't show that? Sorry if the the question looks noob.
top might occasionaly show some...
One source i see for this effect, a modern programming project in C++ with one source file per function and thousands of functions...
Then the CPU is maxed out, and one only sees 5 or so gcc compilers in top with a top cpuload combined of 30%.
ps -ax >t.1
ps -ax >t.2
ps -ax >t.3
diff t.1 t.2
diff t.2 t.3
might show processes that are unique in each..., those are the prime suspects.
top works by sampling the system, and a few seconds later do it again,
all jobs that are missing in either listing are left out on the screen as they have no delta. (and jobs started after the first sample and exited before the next will be missed out entirely).
One source i see for this effect, a modern programming project in C++ with one source file per function and thousands of functions...
Then the CPU is maxed out, and one only sees 5 or so gcc compilers in top with a top cpuload combined of 30%.
ps -ax >t.1
ps -ax >t.2
ps -ax >t.3
diff t.1 t.2
diff t.2 t.3
might show processes that are unique in each..., those are the prime suspects.
top works by sampling the system, and a few seconds later do it again,
all jobs that are missing in either listing are left out on the screen as they have no delta. (and jobs started after the first sample and exited before the next will be missed out entirely).
Premium Content
You need an Expert Office subscription to comment.Start Free Trial