Solved

Xen CPU Load

Posted on 2014-02-18
3
477 Views
1 Endorsement
Last Modified: 2014-02-23
Hi

I am running a Dedicated Server hosting several VPS using SolusVM - XEN PV.

The machine has been running very good for last over a year. Generally the avg CPU load does not even cross 0.5.

Today suddenly the CPU load on the machine has gone crazy. load average: 9.83, 9.48, 9.34

I am not good at finding out which VPS is taking that much resources which made the Dedicated Server go crazy. My colleague who looks after this kind of things isn't available today.

Here is the output of some commands. I will be thankful if someone can help me.

top - 05:47:54 up 298 days,  2:08,  1 user,  load average: 8.72, 9.18, 9.25
Tasks: 230 total,   1 running, 229 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1607680k total,  1329600k used,   278080k free,   108960k buffers
Swap: 16775152k total,   281592k used, 16493560k free,   116116k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                              
 3576 zabbix    22   5 48796 1576 1332 S  0.7  0.1 774:04.70 zabbix_agentd                                                                                                         
    1 root      15   0 10372   84   52 S  0.0  0.0   0:00.18 init                                                                                                                  
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:43.07 migration/0                                                                                                           
    3 root      34  19     0    0    0 S  0.0  0.0  12:14.28 ksoftirqd/0                                                                                                           
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.79 watchdog/0                                                                                                            
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:39.83 migration/1                                                                                                           
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.78 ksoftirqd/1                                                                                                           
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                                                                                                            
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:30.31 migration/2                                                                                                           
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.40 ksoftirqd/2                                                                                                           
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                                                                                                            
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:31.74 migration/3                                                                                                           
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.45 ksoftirqd/3                                                                                                           
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                                                                                                            
   14 root      RT  -5     0    0    0 S  0.0  0.0   0:37.71 migration/4                                                                                                           
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.69 ksoftirqd/4                                                                                                           
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/4                                                                                                            
   17 root      RT  -5     0    0    0 S  0.0  0.0   0:29.38 migration/5                                                                                                           
   18 root      34  19     0    0    0 S  0.0  0.0   0:00.44 ksoftirqd/5                                                                                                           
   19 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/5                                                                                                            
   20 root      RT  -5     0    0    0 S  0.0  0.0   0:28.76 migration/6                                                                                                           
   21 root      34  19     0    0    0 S  0.0  0.0   0:00.43 ksoftirqd/6                                                                                                           
   22 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/6                                                                                                            
   23 root      RT  -5     0    0    0 S  0.0  0.0   0:28.88 migration/7                                                                                                           
   24 root      34  19     0    0    0 S  0.0  0.0   0:00.42 ksoftirqd/7                                                                                                           
   25 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/7                                                                                                            
   26 root      10  -5     0    0    0 S  0.0  0.0   0:21.26 events/0                       

Open in new window


xentop - 05:48:10   Xen 3.4.4
19 domains: 2 running, 17 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 33451412k total, 32916560k used, 534852k free    CPUs: 8 @ 3392MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID
  Domain-0 -----r    1518889    4.8    1607680    4.8   no limit       n/a     8    4        0        0    0        0        0        0    0
     vm102 --b---     792661    2.0    1572864    4.7    1572864       4.7     2    0        0        0    2  6062409 160136545 264895558    0
     vm103 --b---    1462200    6.3    3145728    9.4    3145728       9.4     4    0        0        0    2   913135 57799263 336924845    0
     vm104 -----r     272888    2.6    1572864    4.7    1572864       4.7     2    0        0        0    2   740406 23738563 77958790    0
     vm105 --b---     211172    1.1    1572864    4.7    1572864       4.7     2    0        0        0    2   477369 14770193 57401715    0
     vm107 --b---     716649    1.0    1572864    4.7    1572864       4.7     2    0        0        0    2  3239157 37982829 210484866    0
     vm109 --b---     381013    1.0    1048576    3.1    1048576       3.1     2    0        0        0    2  1259657 49519713 110053341    0
     vm110 --b---     511599    0.8    1572864    4.7    1572864       4.7     2    0        0        0    2  1538311  6910245 109351439    0
     vm112 --b---     481817    1.1    2097152    6.3    2097152       6.3     4    0        0        0    2  1495259 103501955 159422713    0
     vm113 --b---     344605    2.5    1572864    4.7    1572864       4.7     2    0        0        0    2  1567016 48551775 131864182    0
     vm114 --b---     682710    1.2    1572864    4.7    1572864       4.7     2    0        0        0    2  2334762 45116179 207062081    0
     vm115 --b---     149872    0.5    1572864    4.7    1572864       4.7     2    0        0        0    2   609312 11870787 28635900    0
     vm116 --b---     163567    0.4    1048576    3.1    1048576       3.1     2    0        0        0    2   538512   664453 42237896    0
     vm117 --b---     475771    1.6    2621440    7.8    2621440       7.8     4    0        0        0    2  1227400 157533825 150986791    0
     vm121 --b---     446445    1.1    1572864    4.7    1572864       4.7     2    0        0        0    2   868572 19886849 121348440    0
     vm123 --b---     446552    1.1    1572864    4.7    1572864       4.7     2    0        0        0    2  1133882 66091691 129815933    0
     vm125 --b---     424312    1.4    3145728    9.4    3145728       9.4     4    0        0        0    2  2609739 81760057 140218856    0
     vm148 --b---     295968    1.3    1572864    4.7    1572864       4.7     2    0        0        0    2  3121332 57946013 103515546    0
     vm155 --b---      12881    0.0     524288    1.6     524288       1.6     2    0        0        0    2    53082     6540  3035222    0

Open in new window


Total DISK READ: 9.16 M/s | Total DISK WRITE: 1541.08 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                             
16940 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.39 % [blkback.170.xvd]
 7263 be/4 root        0.00 B/s    0.00 B/s  0.39 %  0.39 % mingetty tty1
 2166 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.39 % [kmpathd/0]  
 5426 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.39 % [blkback.5.xvda1]
 6691 be/3 root        2.25 M/s    0.00 B/s  0.00 %  0.39 % [blkback.162.xvd]
12616 be/3 root     1168.55 K/s    0.00 B/s  0.00 %  0.39 % [blkback.173.xvd]
13967 be/3 root       11.76 K/s    0.00 B/s  0.39 %  0.39 % [blkback.184.xvd]
29510 be/4 solusvm     0.00 B/s    0.00 B/s  0.00 %  0.39 % php-cgi
 1042 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kauditd]
 7264 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % mingetty tty2
 5427 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.5.xvda2]
19088 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % dmeventd
 2167 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kmpathd/1]
12617 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.173.xvd]
13968 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.184.xvd]
30060 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % sshd: root@pts/18
15591 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.183.sda]
 6692 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.162.xvd]
21444 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.181.sda]
29142 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.154.sda]
 5635 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.107.sda]
   50 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kblockd/3]
30739 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [md4_resync]
 3004 be/2 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % audispd
  915 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [scsi_eh_1]
   10 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
   13 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
   16 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
   18 be/7 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/5]
   19 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/5]

Open in new window

1
Comment
Question by:sysautomation
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 13

Expert Comment

by:Daniel Helgenberger
ID: 39867005
Hello,

from a first glance on your top output - this could be zabbix? It has a huge CPU time.
Look into Zabbix and see if there are any things wrong; the agent might be crashed.

Also, sort your top by CPU - Time (run it and press '>' until you have the processes sorted by CPU time, or run 'top -o TIME+').

Since monitoring is not essential, try to restart or stop zabbix_agentd on the machine and see if the load goes away; eg. like this:
service zabbix_agent {restart|stop}

Open in new window

0
 

Accepted Solution

by:
sysautomation earned 0 total points
ID: 39867715
I finally found software RAID was synching. I stopped and load came down. Anyways thanks for your help.
0
 

Author Closing Comment

by:sysautomation
ID: 39880409
I finally found software RAID was synching. I stopped and load came down.
0

Featured Post

Get MongoDB database support online, now!

At Percona’s web store you can order your MongoDB database support needs in minutes. No hassles, no fuss, just pick and click. Pay online with a credit card. Handle your MongoDB database support now!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If your vDisk VHD file gets deleted from the image store accidentally or on purpose, you won't be able to remove the vDisk from the PVS console. There is a known workaround that is solid.
When we have a dead host and we lose all connections to the ESXi, and we need to find a way to move all VMs from that dead ESXi host.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

615 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question