Solved

Xen CPU Load

Posted on 2014-02-18
3
435 Views
1 Endorsement
Last Modified: 2014-02-23
Hi

I am running a Dedicated Server hosting several VPS using SolusVM - XEN PV.

The machine has been running very good for last over a year. Generally the avg CPU load does not even cross 0.5.

Today suddenly the CPU load on the machine has gone crazy. load average: 9.83, 9.48, 9.34

I am not good at finding out which VPS is taking that much resources which made the Dedicated Server go crazy. My colleague who looks after this kind of things isn't available today.

Here is the output of some commands. I will be thankful if someone can help me.

top - 05:47:54 up 298 days,  2:08,  1 user,  load average: 8.72, 9.18, 9.25
Tasks: 230 total,   1 running, 229 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1607680k total,  1329600k used,   278080k free,   108960k buffers
Swap: 16775152k total,   281592k used, 16493560k free,   116116k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                              
 3576 zabbix    22   5 48796 1576 1332 S  0.7  0.1 774:04.70 zabbix_agentd                                                                                                         
    1 root      15   0 10372   84   52 S  0.0  0.0   0:00.18 init                                                                                                                  
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:43.07 migration/0                                                                                                           
    3 root      34  19     0    0    0 S  0.0  0.0  12:14.28 ksoftirqd/0                                                                                                           
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.79 watchdog/0                                                                                                            
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:39.83 migration/1                                                                                                           
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.78 ksoftirqd/1                                                                                                           
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                                                                                                            
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:30.31 migration/2                                                                                                           
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.40 ksoftirqd/2                                                                                                           
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                                                                                                            
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:31.74 migration/3                                                                                                           
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.45 ksoftirqd/3                                                                                                           
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                                                                                                            
   14 root      RT  -5     0    0    0 S  0.0  0.0   0:37.71 migration/4                                                                                                           
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.69 ksoftirqd/4                                                                                                           
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/4                                                                                                            
   17 root      RT  -5     0    0    0 S  0.0  0.0   0:29.38 migration/5                                                                                                           
   18 root      34  19     0    0    0 S  0.0  0.0   0:00.44 ksoftirqd/5                                                                                                           
   19 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/5                                                                                                            
   20 root      RT  -5     0    0    0 S  0.0  0.0   0:28.76 migration/6                                                                                                           
   21 root      34  19     0    0    0 S  0.0  0.0   0:00.43 ksoftirqd/6                                                                                                           
   22 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/6                                                                                                            
   23 root      RT  -5     0    0    0 S  0.0  0.0   0:28.88 migration/7                                                                                                           
   24 root      34  19     0    0    0 S  0.0  0.0   0:00.42 ksoftirqd/7                                                                                                           
   25 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/7                                                                                                            
   26 root      10  -5     0    0    0 S  0.0  0.0   0:21.26 events/0                       

Open in new window


xentop - 05:48:10   Xen 3.4.4
19 domains: 2 running, 17 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 33451412k total, 32916560k used, 534852k free    CPUs: 8 @ 3392MHz
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID
  Domain-0 -----r    1518889    4.8    1607680    4.8   no limit       n/a     8    4        0        0    0        0        0        0    0
     vm102 --b---     792661    2.0    1572864    4.7    1572864       4.7     2    0        0        0    2  6062409 160136545 264895558    0
     vm103 --b---    1462200    6.3    3145728    9.4    3145728       9.4     4    0        0        0    2   913135 57799263 336924845    0
     vm104 -----r     272888    2.6    1572864    4.7    1572864       4.7     2    0        0        0    2   740406 23738563 77958790    0
     vm105 --b---     211172    1.1    1572864    4.7    1572864       4.7     2    0        0        0    2   477369 14770193 57401715    0
     vm107 --b---     716649    1.0    1572864    4.7    1572864       4.7     2    0        0        0    2  3239157 37982829 210484866    0
     vm109 --b---     381013    1.0    1048576    3.1    1048576       3.1     2    0        0        0    2  1259657 49519713 110053341    0
     vm110 --b---     511599    0.8    1572864    4.7    1572864       4.7     2    0        0        0    2  1538311  6910245 109351439    0
     vm112 --b---     481817    1.1    2097152    6.3    2097152       6.3     4    0        0        0    2  1495259 103501955 159422713    0
     vm113 --b---     344605    2.5    1572864    4.7    1572864       4.7     2    0        0        0    2  1567016 48551775 131864182    0
     vm114 --b---     682710    1.2    1572864    4.7    1572864       4.7     2    0        0        0    2  2334762 45116179 207062081    0
     vm115 --b---     149872    0.5    1572864    4.7    1572864       4.7     2    0        0        0    2   609312 11870787 28635900    0
     vm116 --b---     163567    0.4    1048576    3.1    1048576       3.1     2    0        0        0    2   538512   664453 42237896    0
     vm117 --b---     475771    1.6    2621440    7.8    2621440       7.8     4    0        0        0    2  1227400 157533825 150986791    0
     vm121 --b---     446445    1.1    1572864    4.7    1572864       4.7     2    0        0        0    2   868572 19886849 121348440    0
     vm123 --b---     446552    1.1    1572864    4.7    1572864       4.7     2    0        0        0    2  1133882 66091691 129815933    0
     vm125 --b---     424312    1.4    3145728    9.4    3145728       9.4     4    0        0        0    2  2609739 81760057 140218856    0
     vm148 --b---     295968    1.3    1572864    4.7    1572864       4.7     2    0        0        0    2  3121332 57946013 103515546    0
     vm155 --b---      12881    0.0     524288    1.6     524288       1.6     2    0        0        0    2    53082     6540  3035222    0

Open in new window


Total DISK READ: 9.16 M/s | Total DISK WRITE: 1541.08 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                             
16940 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.39 % [blkback.170.xvd]
 7263 be/4 root        0.00 B/s    0.00 B/s  0.39 %  0.39 % mingetty tty1
 2166 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.39 % [kmpathd/0]  
 5426 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.39 % [blkback.5.xvda1]
 6691 be/3 root        2.25 M/s    0.00 B/s  0.00 %  0.39 % [blkback.162.xvd]
12616 be/3 root     1168.55 K/s    0.00 B/s  0.00 %  0.39 % [blkback.173.xvd]
13967 be/3 root       11.76 K/s    0.00 B/s  0.39 %  0.39 % [blkback.184.xvd]
29510 be/4 solusvm     0.00 B/s    0.00 B/s  0.00 %  0.39 % php-cgi
 1042 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kauditd]
 7264 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % mingetty tty2
 5427 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.5.xvda2]
19088 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % dmeventd
 2167 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kmpathd/1]
12617 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.173.xvd]
13968 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.184.xvd]
30060 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % sshd: root@pts/18
15591 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.183.sda]
 6692 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.162.xvd]
21444 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.181.sda]
29142 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.154.sda]
 5635 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [blkback.107.sda]
   50 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kblockd/3]
30739 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [md4_resync]
 3004 be/2 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % audispd
  915 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [scsi_eh_1]
   10 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
   13 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
   16 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
   18 be/7 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/5]
   19 rt/3 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/5]

Open in new window

1
Comment
Question by:sysautomation
  • 2
3 Comments
 
LVL 13

Expert Comment

by:Daniel Helgenberger
ID: 39867005
Hello,

from a first glance on your top output - this could be zabbix? It has a huge CPU time.
Look into Zabbix and see if there are any things wrong; the agent might be crashed.

Also, sort your top by CPU - Time (run it and press '>' until you have the processes sorted by CPU time, or run 'top -o TIME+').

Since monitoring is not essential, try to restart or stop zabbix_agentd on the machine and see if the load goes away; eg. like this:
service zabbix_agent {restart|stop}

Open in new window

0
 

Accepted Solution

by:
sysautomation earned 0 total points
ID: 39867715
I finally found software RAID was synching. I stopped and load came down. Anyways thanks for your help.
0
 

Author Closing Comment

by:sysautomation
ID: 39880409
I finally found software RAID was synching. I stopped and load came down.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

VMware Update Manager(VUM) “error code: 15” during ESXi 6.0 Remediate update in VUM operation
This is similar to the error described in my previous Article, but with a different source problem and a different solution. When trying to scan and stage the ESXi 6.0 updates using VMware Update Manager, we can get "error code 90"
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now