?
Solved

Server lagging at peak times

Posted on 2003-03-19
8
Medium Priority
?
418 Views
Last Modified: 2012-05-04
My server is lagging at peak times. In the morning it is pretty fast.
Here is a top -i @ 12.40est march-19

 12:46pm  up 8 days, 13:41,  1 user,  load average: 24.17, 23.87, 18.49
362 processes: 322 sleeping, 40 running, 0 zombie, 0 stopped
CPU states: 58.4% user, 23.7% system, 15.0% nice,  2.7% idle
Mem:   998872K av,  987360K used,   11512K free,       0K shrd,    5488K buff
Swap: 2096472K av,   27540K used, 2068932K free                  392176K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 5500 root      15   0  1324 1324   832 R    12.2  0.1   0:09 top
 3725 mysql     26  10  116M 116M  2232 R N   0.8 11.9   0:09 mysqld
 3819 mysql     25  10  116M 116M  2232 R N   0.6 11.9   0:06 mysqld
 3665 mysql     26  10  116M 116M  2232 R N   0.4 11.9   0:05 mysqld
 3749 mysql     25  10  116M 116M  2232 R N   0.4 11.9   0:04 mysqld
 3808 mysql     25  10  116M 116M  2232 R N   0.4 11.9   0:02 mysqld
 3821 mysql     25  10  116M 116M  2232 R N   0.3 11.9   0:03 mysqld
 4750 mysql     25  10  116M 116M  2232 R N   0.3 11.9   0:03 mysqld
 4756 mysql     26  10  116M 116M  2232 R N   0.3 11.9   0:00 mysqld
 4977 mysql     25  10  116M 116M  2232 R N   0.3 11.9   0:02 mysqld
 3671 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:05 mysqld
 3684 mysql     26  10  116M 116M  2232 R N   0.2 11.9   0:09 mysqld
 3719 mysql     26  10  116M 116M  2232 R N   0.2 11.9   0:06 mysqld
 3726 mysql     26  10  116M 116M  2232 R N   0.2 11.9   0:06 mysqld
 3738 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:04 mysqld
 3739 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:05 mysqld
 3764 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:07 mysqld
 3782 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:03 mysqld
 4975 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:01 mysqld
 3673 mysql     25  10  116M 116M  2232 R N   0.1 11.9   0:02 mysqld
 3703 mysql     26  10  116M 116M  2232 R N   0.1 11.9   0:06 mysqld
 3714 mysql     25  10  116M 116M  2232 R N   0.1 11.9   0:06 mysqld
 4751 mysql     25  10  116M 116M  2232 R N   0.1 11.9   0:02 mysqld
 3464 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:01 mysqld
 3661 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:00 mysqld
 3670 mysql     26  10  116M 116M  2232 R N   0.0 11.9   0:03 mysqld
 3677 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:06 mysqld
 3678 mysql     26  10  116M 116M  2232 R N   0.0 11.9   0:02 mysqld
 3683 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:05 mysqld
 3720 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:08 mysqld
 3732 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:01 mysqld
 3761 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:02 mysqld
 3788 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:03 mysqld
 3810 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:06 mysqld
 3827 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:03 mysqld
 3829 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:10 mysqld
 4748 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:04 mysqld
 4749 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:02 mysqld
 5026 mysql     25  10  117M 117M  2232 R N   0.0 12.0   0:01 mysqld
 5517 apache    15   0  7296 4992  4684 R     0.0  0.4   0:00 httpd

This is driving me insane and need toget it fixed. it is a Celeron 1.7 1gb ram running vBulletin.

I did a ps xa and got this...

  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:06 init [3]
    2 ?        SW     0:00 [keventd]
    3 ?        SW     0:04 [kapmd]
    4 ?        RWN    0:00 [ksoftirqd_CPU0]
    5 ?        SW     2:48 [kswapd]
    6 ?        SW     0:13 [bdflush]
    7 ?        SW     0:01 [kupdated]
    8 ?        SW     0:00 [mdrecoveryd]
   12 ?        SW     1:54 [kjournald]
   92 ?        SW     0:00 [khubd]
  253 ?        SW     0:00 [kjournald]
  828 ?        SW     0:00 [eth0]
 1042 ?        S      0:00 portmap
 1070 ?        S      0:00 rpc.statd
 1183 ?        S      0:33 syslogd -m 0 -a /home/virtual/FILESYSTEMTEMPLATE/log-
 1197 ?        S      0:03 klogd
 1284 ?        S      0:08 /usr/sbin/sshd
 1304 ?        S      0:00 sshd -f /etc/ssh/sshd-rb_config
 1338 ?        S      0:01 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid
 1567 ?        S      0:13 /usr/sbin/bandwidth_manager
 1620 ?        S      0:00 /usr/sbin/atd
 1672 ?        S      0:09 /usr/bin/perl -w /usr/sbin/kmsgsd
 1681 tty1     S      0:00 /sbin/mingetty tty1
 1682 tty2     S      0:00 /sbin/mingetty tty2
 1683 tty3     S      0:00 /sbin/mingetty tty3
 1684 tty4     S      0:00 /sbin/mingetty tty4
 1685 tty5     S      0:00 /sbin/mingetty tty5
 1686 tty6     S      0:00 /sbin/mingetty tty6
 4672 ?        S      0:30 /usr/local/jabber/jabber-1.4.2/jabberd/jabberd
 4673 ?        S      0:00 /usr/local/jabber/jabber-1.4.2/jabberd/jabberd
 4697 ?        S      0:07 sendmail: accepting connections                      
 4720 ?        S      0:00 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailSc
22898 ?        S<     0:00 named -u named
22901 ?        S<     0:00 named -u named
22906 ?        S<     0:00 named -u named
22907 ?        S<     0:00 named -u named
22908 ?        S<     0:00 named -u named
 5169 ?        S      0:00 /usr/bin/postmaster -D /var/lib/pgsql/data
 5490 ?        S      0:05 postgres: root appldb [local] idle        
 6586 ?        S      0:51 ./bwbar --output-file ../bandwidth/index.shtml --time
 7267 ?        S      0:01 proftpd: (accepting connections)
10556 ?        S      0:00 /bin/sh /usr/bin/safe_mysqld --defaults-file=/etc/my.
 2086 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2087 ?        S<     0:00 /usr/sbin/fcgi-pm  -DSSL -d /usr/lib/opcenter -f fast
 2088 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2089 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2090 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2091 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2092 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2093 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
 2453 ?        S      0:45 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2462 ?        S     11:16 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2464 ?        S     11:23 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2465 ?        S     11:32 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2466 ?        S     10:38 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2467 ?        S     11:36 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2470 ?        S     12:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
16035 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
16036 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16037 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16038 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16039 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16040 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16041 ?        S<     0:01 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
21075 ?        S      6:40 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
26904 ?        S      4:51 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
26905 ?        S      4:28 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30035 ?        S      2:35 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30036 ?        S      2:48 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30037 ?        S      3:16 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30230 ?        S      2:38 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30231 ?        S      3:21 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30232 ?        S      2:48 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30618 ?        S      2:55 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30620 ?        S      2:37 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31446 ?        S      2:16 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31448 ?        S      2:07 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31632 ?        S      2:22 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31791 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
  431 ?        S      1:41 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  432 ?        S      1:37 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  434 ?        S      1:26 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  464 ?        S      1:33 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  465 ?        S      1:34 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  467 ?        S      1:21 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 1506 ?        S      1:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2313 ?        S      0:41 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2315 ?        S      0:26 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2497 ?        S      0:45 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2498 ?        S      0:44 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 3246 ?        S      0:26 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 3464 ?        RN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3661 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3662 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3663 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3664 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3665 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3666 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3667 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3668 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3669 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3670 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3671 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3672 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3673 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3674 ?        RN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3675 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3676 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3677 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3678 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3679 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3680 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3681 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3682 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3683 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3684 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3685 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3686 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3687 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3688 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3689 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3690 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3691 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3692 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3693 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3694 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3695 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3696 ?        SN     0:09 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3697 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3698 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3699 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3700 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3701 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3702 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3703 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3704 ?        RN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3705 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3706 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3707 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3708 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3709 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3710 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3711 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3712 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3713 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3714 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3715 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3716 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3717 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3718 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3719 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3720 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3721 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3722 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3723 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3724 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3725 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3726 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3727 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3728 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3729 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3730 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3731 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3732 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3733 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3734 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3735 ?        SN     0:09 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3736 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3737 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3738 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3739 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3741 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3748 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3749 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3750 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3751 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3760 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3761 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3762 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3763 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3764 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3765 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3766 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3767 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3768 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3769 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3773 ?        SN     0:11 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3774 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3775 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3777 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3780 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3781 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3782 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3787 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3788 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3789 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3798 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3799 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3800 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3801 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3802 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3803 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3804 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3805 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3806 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3807 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3808 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3809 ?        RN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3810 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3811 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3812 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3813 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3814 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3815 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3816 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3817 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3818 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3819 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3820 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3821 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3822 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3823 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3824 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3825 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3826 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3827 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3828 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3829 ?        SN     0:09 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3934 ?        S      0:19 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4202 ?        S      0:02 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailSc
 4270 ?        S      0:17 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4435 ?        S      0:10 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4440 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4441 ?        S      0:08 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4442 ?        S      0:10 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4444 ?        S      0:07 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4445 ?        S      0:12 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4446 ?        S      0:17 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4468 ?        S      0:13 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4491 ?        S      0:10 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4493 ?        S      0:15 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4515 ?        S      0:12 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4517 ?        S      0:11 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4518 ?        S      0:13 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4598 ?        S      0:08 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4602 ?        S      0:11 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4609 ?        S      0:09 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4645 ?        S      0:03 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4747 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4748 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4749 ?        RN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4750 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4751 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4752 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4753 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4754 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4755 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4756 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4757 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4758 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4759 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4862 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4863 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4879 ?        S      0:04 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4880 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4881 ?        S      0:07 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4882 ?        S      0:04 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4884 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4909 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4975 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4976 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4977 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4978 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4979 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4980 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4981 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4982 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4983 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4984 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4985 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5017 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5018 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5019 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5020 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5021 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5022 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5023 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5024 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5025 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5026 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5028 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5044 ?        S      0:03 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5136 ?        S      0:00 /usr/sbin/sshd
 5138 pts/1    S      0:00 -bash
 5192 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5193 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5194 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5195 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5196 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5197 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5198 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5199 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5200 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5201 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5202 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5203 ?        S      0:02 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5204 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5207 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5209 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5271 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5300 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5301 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5302 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5305 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5309 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5319 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5320 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5322 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5324 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5325 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5327 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5328 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5329 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5330 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5331 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5332 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS


I really hope someone can help me out here.

THanks,
Derek


0
Comment
Question by:7thgencivic
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
8 Comments
 
LVL 4

Expert Comment

by:gletiecq
ID: 8169208
A couple of comments...

You may want to segregate your database server and your http server.  Since the activity on each is related to the activity of the other, when one is clamoring for resources that's exactly the time when the other is as well.

You're running some apps that probably don't belong on that server, as it means you're opening virtually all of the ports on that box.  Sendmail, named, postgres, ftp, your python scripts, and everything else may better belong somewhere else.  If you want to secure a DB server or an http server, it's a lot easier when you lock down all the other stuff and move those other functions to another box.  Having everything on one box makes it hard to determine what can be locked down and hard to spot when you've got problems.

Google for async threading.  It looks like you're running multiple sync threads per process, and that doesn't scale too well above 500 or so users.  I think there's a recent article on osopinions.com about this where they look at this issue in WinXP, RH and Solaris.

Check out that the cache+free memory looks like when things are running smoothly versus when things are bogging down.  You've got a fair amount of memory on that box, so I don't suspect you're starving for more, but if you see a huge difference between these times it's probably worth throwing some more memory at the problem.

Hope this helps.

Greg

0
 

Author Comment

by:7thgencivic
ID: 8169511
I am running an ensim box, this is why all those are running. I have talked to ppl on forums.rackshack.net and ppl have the same box pretty much, or less and more users and it barely makes a dent.
On a max, I will get 250 concurrent users.

Derek
0
 
LVL 20

Expert Comment

by:Gns
ID: 8174767
Your load average is kind of high (load average: 24.17, 23.87, 18.49) for a one CPU system, but you seem to have plenty of memory left (free+cached ~400MiB), so ... I would have guessed that you suffered from an IO bootleneck "somewhere".

One reason for high load averages is that processes in state D add 1 each to the average, but that should have shown up in the ps listing... hmmm.

I'd still look long and hard on the NIC driver and the disk subsystem. Some NIC drivers spend an inordinate time in kernel.

Gregs advice about dividing the workload on more machines is good (I'd go for at least 3... 1 web server, 1 database server and 1 for the rest).

-- Glenn
0
Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.

 

Author Comment

by:7thgencivic
ID: 8175066
Thanks Glenn, but that is not practical given my situation.
I may be able to do 2 boxes, but now more. THere has to be something that I am able to do, given the fact that other ppl have the same type of setup and have no issues like this?

Derek
0
 
LVL 20

Expert Comment

by:Gns
ID: 8175372
Yes, start by looking at getting a "baseline" for normal operation with regards to memory status (vmstat, top sorted by (M)emory ...), processes, IO stats etc. the sar package should be useable, albeit a bit rudimentary:-).
When you have that, you will probably see where the problem areas are. Booring but effective.

Another thing that can drive the load average through the roof, without leaving much trace, is a fork-bomb or processes behaving in the same manner (spawning rather fast, but perhaps not living so long that you "register" them).
If you note a very rapid progression in process IDs, this might be it.

Say that some bozo (I had a chineese host attact our mail-server today...) really stresses out sendmail, or rather your mailfilter.... could be bad.

I recommend the swordfish book ("System Performance Tuning, 2nd edition" from Oreilly) if you are interrested in some in-depth thoughts on performance tuning.

-- Glenn
0
 
LVL 20

Expert Comment

by:Gns
ID: 8175397
Is that ps listing from the same time as the top?

-- Glenn
0
 

Author Comment

by:7thgencivic
ID: 8175876
no it is not.
0
 
LVL 20

Accepted Solution

by:
Gns earned 800 total points
ID: 8179586
OK, then we cannot right off there being a lot of processes in state D (non-interruptible wait-state, which as you know is most often waiting for IO to complete in some form (often disk or network IO)).
When you have these exorbitant loads, do
ps auxww|grep " D "
or just page through the listing, and make a note on which/how many processes is in state D.
If there are a fair amount (20+) and they seem to be of the same "type" (mostly network (http, mail etc.) or mostly disk (mysql ... perhaps the mailfilter)), you've got a lead on where the (probable) bottleneck is.

Getting the "baseline" will make it easier to "see" the extraordinary conditions.

Good Luck
-- Glenn
0

Featured Post

WordPress Tutorial 4: Recommended Plugins

Now that you have WordPress installed, understand the interface, and know how to install new parts, let’s take a look at our recommended plugins.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month14 days, 12 hours left to enroll

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question