High CPU Load

HI

I am running Xen-PV with SolusVM on Centos 6 and hosting about 11 Virtual machines. 5. Since yesterday the cpu load on my host machine goes very high and reaches between 28-35. Connecting with SSH becomes very hard. I run top command but did not find any processes taking very high CPU.
I am puzzled. I also ran xm top command (screenshot attached).
Also /var/log/messages show below message:

Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: audit_lost=150 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: audit_lost=151 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: audit_lost=152 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:01 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:01 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit_log_start: 2 callbacks suppressed
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: audit_lost=154 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: audit_lost=155 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: audit_lost=156 audit_rate_limit=0 audit_backlog_limit=320
Oct 19 16:47:20 slave-wahad kernel: audit: backlog limit exceeded
Oct 19 16:47:20 slave-wahad kernel: audit: audit_backlog=321 > audit_backlog_limit=320

Open in new window

and

# aureport --start today --event --summary -i

Event Summary Report
======================
total  type
======================
6751  NETFILTER_CFG
1470  USER_START
1462  LOGIN
1462  USER_ACCT
1462  CRED_ACQ
1446  USER_END
1445  CRED_DISP
1415  CRYPTO_KEY_USER
748  USER_AUTH
584  CRYPTO_SESSION
538  USER_LOGIN
265  USER_ERR
11  ANOM_PROMISCUOUS
8  CRED_REFR
2  USER_LOGOUT
1  CONFIG_CHANGE

Open in new window

I would be really thankful if someone can please help me identify what is causing this high CPU load.
Screen-Shot-2015-10-19-at-8.42.17-PM.png

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

LVL 46

nociSoftware EngineerCommented: 2015-10-20

if you have a lot of short lived processes, they may not show up in top, while causing a boatload of CPU usage.
Try to find out if there are some runaway server restarts...

The problem causing processes might also cause a lot of logging, that may help. Look for processes with a lot of fast changing pid's

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial

sysautomationAuthor Commented: 2015-10-20

Thanks for the reply.

> The problem causing processes might also cause a lot of logging, that may help. Look for processes with a lot of fast changing pid's

How can I see those processes when top doesn't show that? Sorry if the the question looks noob.

nociSoftware EngineerCommented: 2015-10-21

top might occasionaly show some...
One source i see for this effect, a modern programming project in C++ with one source file per function and thousands of functions...

Then the CPU is maxed out, and one only sees 5 or so gcc compilers in top with a top cpuload combined of 30%.

ps -ax >t.1
ps -ax >t.2
ps -ax >t.3

diff t.1 t.2
diff t.2 t.3

might show processes that are unique in each..., those are the prime suspects.

top works by sampling the system, and a few seconds later do it again,
all jobs that are missing in either listing are left out on the screen as they have no delta. (and jobs started after the first sample and exited before the next will be missed out entirely).

High CPU Load

Who is Participating?