Avoiding CPU speed scaling in modern Linux distributions. Running CPU at full speed Tips.

AID: 3492
  • Status: Published

4790 points

  • ByNopius
  • TypeTips/Tricks
  • Posted on2010-07-26 at 07:12:48
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power.

Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on lower CPU speed, then they could. This can be easily checked with this command:

grep -E '^model name|^cpu MHz' /proc/cpuinfo
                                    
1:

Select allOpen in new window



What you can see:

model name      : Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
cpu MHz         : 1600.000
...
                                    
1:
2:
3:

Select allOpen in new window


or
model name      : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
cpu MHz          : 1596.000
...
                                    
1:
2:
3:

Select allOpen in new window


or even
model name      : Intel(R) Core(TM)2 CPU          4400  @ 2.00GHz
cpu MHz         : 1000.000
model name      : Intel(R) Core(TM)2 CPU          4400  @ 2.00GHz
cpu MHz         : 600.000
                                    
1:
2:
3:
4:

Select allOpen in new window



Oops, we are paying for 2 Core 2Ghz CPU that runs on 600Mhz on one core and 1000Mhz on another!!!

There will be other lines for all the CPUs/cores/threads, probably with the same values.

This feature is nice, if we are running workstation, but what I've noticed, we do have the same CPU throttling on Ubuntu Server 10.04 builds and on CentOS 5.3, 5.4 and 5.5 builds (thus on RedHat too).

After hours of digging google, I've found that:
- this problem is very common
- there are several bug reports about this issue
- this is not BIOS settings problem, because on dual boot systems, CPU runs at full speed on Windows
- there are no 100% working solutions or they are too difficult to find
- this is not a bug, but a 'feature' of the new kernels, it is implemented differently on 2.6.18 (CentOS) and 2.6.32 (Ubuntu).

Here is a tip how to disable it on running system:

1) Check that 'kondemand' thread is running, run as root:  "pgrep -lf ondemand"

the output should be like:
[root@boston07 ~]# uname -a
Linux boston07 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:18:27 EST 2009 i686 i686 i386 GNU/Linux
[root@boston07 ~]# pgrep -lf ondemand
1444 kondemand/0
1445 kondemand/1
                                    
1:
2:
3:
4:
5:

Select allOpen in new window



2) Check that current cpu speed differs from the maximum:

[root@boston07 ~]# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 1596.000
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 1596.000
                                    
1:
2:
3:
4:
5:

Select allOpen in new window



3) Change CPU governor from 'ondemand' to 'performance' for all CPUs/cores, run as root:
for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f $CPUFREQ ] || continue; echo -n performance > $CPUFREQ; done
                                    
1:

Select allOpen in new window



4) Check that your changes have been applied:

[root@boston07 ~]# grep -E '^model name|^cpu MHz' /proc/cpuinfo
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 2394.000
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
cpu MHz         : 2394.000
                                    
1:
2:
3:
4:
5:

Select allOpen in new window



5) If you are running 'cpuspeed', 'cpufreqd', 'powerd' or other daemons, that can control CPU stepping, just stop them, if you really need to run your system on 100% of the CPU speed.
On CentOS:
# service cpuspeed stop
                                    
1:

Select allOpen in new window



6) On Linux 2.6.32 (On RedHat 6, and Oracle Unbreakable Linux 6) remove CPU scaling kernel modules:
# lsmod | grep ondemand
cpufreq_ondemand        8764  0 
freq_table              3751  2 cpufreq_ondemand,acpi_cpufreq
# rmmod cpufreq_ondemand acpi_cpufreq freq_table
                                    
1:
2:
3:
4:

Select allOpen in new window



Ensure that no 'kondemand' kernel threads are running:
# pgrep -lf kondemand
#
                                    
1:
2:

Select allOpen in new window



7) To make changes permanent (on reboot):

- On Ubuntu, modify /etc/init.d/ondemand script:

change this
echo -n ondemand > $CPUFREQ
                                    
1:

Select allOpen in new window


to this:
echo -n performance > $CPUFREQ
                                    
1:

Select allOpen in new window



OR ALTERNATIVELY just remove all references to ondemand from /etc/rc?.d/
rm -f /etc/rc?.d/S99ondemand
                                    
1:

Select allOpen in new window


- On CentOS, just create a new script /etc/init.d/ondemand:
#! /bin/bash
#
# ondemand sets cpu govermor
#
# chkconfig: 2345 10 90
#
# description: Set the CPU Frequency Scaling governor to "performance"
#
### BEGIN INIT INFO
# Provides: $ondemand
### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin

case "$1" in
    start)
        for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
        do
                [ -f $CPUFREQ ] || continue
                echo -n performance > $CPUFREQ
        done
        ;;
    restart|reload|force-reload)
        echo "Error: argument '$1' not supported" >&2
        exit 3
        ;;
    stop)
        ;;
    *)
        echo "Usage: $0 start|stop" >&2
        exit 3
        ;;
esac
                                    
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:

Select allOpen in new window


then enable it:
chmod +x /etc/init.d/ondemand
chkconfig --add ondemand
service ondemand restart
                                    
1:
2:
3:

Select allOpen in new window



I'm using 'ondemand' name of the script, this may be a little bit misleading (because really it is a 'performance'), but you may change it.

Here are some useful links (just FYI, don't try to install cpufreq-set, cpufreq-get or other utilities, they will also install themselves as daemons, that will control your cpu speed regardless of your cpu governor settings):

1.  http://www.redhat.com/docs/wp/performancetuning/powermanagement/cpufreq_governors.html

2. http://ego.randomwalk.in/blog/2008/04/04/cause-and-effect/


Regards,
Arty
Asked On
2010-07-26 at 07:12:48ID3492
Tags

cpufreq

,

slow speed

,

/proc/cpuinfo

,

ondemand

,

cpuspeed

,

cpufreq-set

,

speedstep

,

kondemand

,

speedup

,

speed scaling

,

cpu throttle

Topic

Linux Administration

Views
3816

Comments

Expert Comment

by: jjmcd on 2011-02-03 at 06:16:56ID: 23448

I'm not sure I understand the use case where you wish to maximize your electric bill.

With Ondemand, the CPU runs faster when it needs to, slower when it doesn't need to.  This results in lower electrical use and temperatures, hence longer CPU life and lower electric and cooling bills.

If you do the "grep ... cpuinfo" enough times you will see several different results (presuming that your system actually has some sort of load).  Better yet, add the cpu speed monitor to the panel so you can see it change as you do stuff.  Of course, if your system isn't doing anything then usually it will show the lowest speed, while if it is fully loaded, it will almost always show the maximum speed.  Your  "grep ... cpuinfo" simply takes a snapshot, which may or may not be representative ... it's simply a roll of the dice.

For most workloads, Ondemand represents a nice balance between performance and energy use..  Performance supposedly leans towards performance a little more, but I have never been able to see a meaningful difference.

Simply locking the CPU speed at the max only means that you run the idle loop as fast as possible, while increasing CPU power consumption, CPU temperature and fan speed, while reducing the life of your CPU.

Personally, I've never been that concerned about seeing the idle loop run as fast as possible.

Author Comment

by: Nopius on 2011-02-03 at 06:35:57ID: 23449

When you rent a server in a datacenter, that doesn't matter how much it comsumes - you pay fixed monthly fee.

Sometimes this matters how FAST your EVERY request is processed regardless of CPU load average.

Here is an example.
There are ping lowering services, like smoothping.com,where every milisecond of packet processing matters (users run ssh, that connects to server, then they use socks tunnel, provided by openssh, that helps to reduce ping, say from 300ms to 150ms, that is very important in online games).
If your load average is below 0.2 (even Atom 1.6Ghz can manage 100 ssh users with 0.2 LA), then your CPU runs on, say 600Mhz instead of 1.6Ghz, that makes every user connection a bit slower (if you don't beleave me - you can run tests). Does that make sense?

Regards,
Arty

Add your Comment

Please Sign up or Log in to comment on this article.

Join Experts Exchange Today

Gain Access to all our Tech Resources

Get personalized answers

Ask unlimited questions

Access Proven Solutions

Search 3.2 million solutions

Read In-Depth How-To Guides

1000+ articles, demos, & tips

Watch Step by Step Tutorials

Learn direct from top tech pros

And Much More!

Your complete tech resource

See Plans and Pricing

30-day free trial. Register in 60 seconds.

Loading Advertisement...

Top Linux Admin Experts

  1. maeltar

    4,300

    0 points yesterday

    Profile
    Rank: Guru
  2. xterm

    4,000

    0 points yesterday

    Profile
    Rank: Sage
  3. noci

    2,800

    0 points yesterday

    Profile
    Rank: Genius
  4. duncan_roe

    2,800

    0 points yesterday

    Profile
    Rank: Genius
  5. pcsmitpra

    1,800

    0 points yesterday

    Profile
    Rank: Wizard
  6. testez

    1,800

    0 points yesterday

    Profile
    Rank: Guru
  7. Nopius

    1,470

    30 points yesterday

    Profile
    Rank: Genius
  8. abbright

    1,000

    0 points yesterday

    Profile
    Rank: Guru
  9. torakeshb

    1,000

    0 points yesterday

    Profile
    Rank: Master
  10. woolmilkporc

    800

    0 points yesterday

    Profile
    Rank: Genius
  11. dbauermann

    600

    0 points yesterday

    Profile
  12. wasimibm

    600

    0 points yesterday

    Profile
    Rank: Guru
  13. unassassinable

    420

    0 points yesterday

    Profile
    Rank: Guru
  14. Anton74

    390

    0 points yesterday

    Profile
    Rank: Guru
  15. kosarajudeepak

    340

    0 points yesterday

    Profile
    Rank: Wizard
  16. abolinhas

    320

    0 points yesterday

    Profile
    Rank: Guru
  17. viju2008

    300

    0 points yesterday

    Profile
    Rank: Master
  18. farzanj

    300

    0 points yesterday

    Profile
    Rank: Genius
  19. OP_Zaharin

    300

    0 points yesterday

    Profile
    Rank: Sage
  20. ahmedhamdy27

    290

    0 points yesterday

    Profile
  21. allen-davis

    280

    10 points yesterday

    Profile
    Rank: Master

Hall Of Fame