• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 419
  • Last Modified:

Server lagging at peak times

My server is lagging at peak times. In the morning it is pretty fast.
Here is a top -i @ 12.40est march-19

 12:46pm  up 8 days, 13:41,  1 user,  load average: 24.17, 23.87, 18.49
362 processes: 322 sleeping, 40 running, 0 zombie, 0 stopped
CPU states: 58.4% user, 23.7% system, 15.0% nice,  2.7% idle
Mem:   998872K av,  987360K used,   11512K free,       0K shrd,    5488K buff
Swap: 2096472K av,   27540K used, 2068932K free                  392176K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 5500 root      15   0  1324 1324   832 R    12.2  0.1   0:09 top
 3725 mysql     26  10  116M 116M  2232 R N   0.8 11.9   0:09 mysqld
 3819 mysql     25  10  116M 116M  2232 R N   0.6 11.9   0:06 mysqld
 3665 mysql     26  10  116M 116M  2232 R N   0.4 11.9   0:05 mysqld
 3749 mysql     25  10  116M 116M  2232 R N   0.4 11.9   0:04 mysqld
 3808 mysql     25  10  116M 116M  2232 R N   0.4 11.9   0:02 mysqld
 3821 mysql     25  10  116M 116M  2232 R N   0.3 11.9   0:03 mysqld
 4750 mysql     25  10  116M 116M  2232 R N   0.3 11.9   0:03 mysqld
 4756 mysql     26  10  116M 116M  2232 R N   0.3 11.9   0:00 mysqld
 4977 mysql     25  10  116M 116M  2232 R N   0.3 11.9   0:02 mysqld
 3671 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:05 mysqld
 3684 mysql     26  10  116M 116M  2232 R N   0.2 11.9   0:09 mysqld
 3719 mysql     26  10  116M 116M  2232 R N   0.2 11.9   0:06 mysqld
 3726 mysql     26  10  116M 116M  2232 R N   0.2 11.9   0:06 mysqld
 3738 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:04 mysqld
 3739 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:05 mysqld
 3764 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:07 mysqld
 3782 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:03 mysqld
 4975 mysql     25  10  116M 116M  2232 R N   0.2 11.9   0:01 mysqld
 3673 mysql     25  10  116M 116M  2232 R N   0.1 11.9   0:02 mysqld
 3703 mysql     26  10  116M 116M  2232 R N   0.1 11.9   0:06 mysqld
 3714 mysql     25  10  116M 116M  2232 R N   0.1 11.9   0:06 mysqld
 4751 mysql     25  10  116M 116M  2232 R N   0.1 11.9   0:02 mysqld
 3464 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:01 mysqld
 3661 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:00 mysqld
 3670 mysql     26  10  116M 116M  2232 R N   0.0 11.9   0:03 mysqld
 3677 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:06 mysqld
 3678 mysql     26  10  116M 116M  2232 R N   0.0 11.9   0:02 mysqld
 3683 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:05 mysqld
 3720 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:08 mysqld
 3732 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:01 mysqld
 3761 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:02 mysqld
 3788 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:03 mysqld
 3810 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:06 mysqld
 3827 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:03 mysqld
 3829 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:10 mysqld
 4748 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:04 mysqld
 4749 mysql     25  10  116M 116M  2232 R N   0.0 11.9   0:02 mysqld
 5026 mysql     25  10  117M 117M  2232 R N   0.0 12.0   0:01 mysqld
 5517 apache    15   0  7296 4992  4684 R     0.0  0.4   0:00 httpd

This is driving me insane and need toget it fixed. it is a Celeron 1.7 1gb ram running vBulletin.

I did a ps xa and got this...

  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:06 init [3]
    2 ?        SW     0:00 [keventd]
    3 ?        SW     0:04 [kapmd]
    4 ?        RWN    0:00 [ksoftirqd_CPU0]
    5 ?        SW     2:48 [kswapd]
    6 ?        SW     0:13 [bdflush]
    7 ?        SW     0:01 [kupdated]
    8 ?        SW     0:00 [mdrecoveryd]
   12 ?        SW     1:54 [kjournald]
   92 ?        SW     0:00 [khubd]
  253 ?        SW     0:00 [kjournald]
  828 ?        SW     0:00 [eth0]
 1042 ?        S      0:00 portmap
 1070 ?        S      0:00 rpc.statd
 1183 ?        S      0:33 syslogd -m 0 -a /home/virtual/FILESYSTEMTEMPLATE/log-
 1197 ?        S      0:03 klogd
 1284 ?        S      0:08 /usr/sbin/sshd
 1304 ?        S      0:00 sshd -f /etc/ssh/sshd-rb_config
 1338 ?        S      0:01 xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid
 1567 ?        S      0:13 /usr/sbin/bandwidth_manager
 1620 ?        S      0:00 /usr/sbin/atd
 1672 ?        S      0:09 /usr/bin/perl -w /usr/sbin/kmsgsd
 1681 tty1     S      0:00 /sbin/mingetty tty1
 1682 tty2     S      0:00 /sbin/mingetty tty2
 1683 tty3     S      0:00 /sbin/mingetty tty3
 1684 tty4     S      0:00 /sbin/mingetty tty4
 1685 tty5     S      0:00 /sbin/mingetty tty5
 1686 tty6     S      0:00 /sbin/mingetty tty6
 4672 ?        S      0:30 /usr/local/jabber/jabber-1.4.2/jabberd/jabberd
 4673 ?        S      0:00 /usr/local/jabber/jabber-1.4.2/jabberd/jabberd
 4697 ?        S      0:07 sendmail: accepting connections                      
 4720 ?        S      0:00 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailSc
22898 ?        S<     0:00 named -u named
22901 ?        S<     0:00 named -u named
22906 ?        S<     0:00 named -u named
22907 ?        S<     0:00 named -u named
22908 ?        S<     0:00 named -u named
 5169 ?        S      0:00 /usr/bin/postmaster -D /var/lib/pgsql/data
 5490 ?        S      0:05 postgres: root appldb [local] idle        
 6586 ?        S      0:51 ./bwbar --output-file ../bandwidth/index.shtml --time
 7267 ?        S      0:01 proftpd: (accepting connections)
10556 ?        S      0:00 /bin/sh /usr/bin/safe_mysqld --defaults-file=/etc/my.
 2086 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2087 ?        S<     0:00 /usr/sbin/fcgi-pm  -DSSL -d /usr/lib/opcenter -f fast
 2088 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2089 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2090 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2091 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2092 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
 2093 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
 2453 ?        S      0:45 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2462 ?        S     11:16 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2464 ?        S     11:23 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2465 ?        S     11:32 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2466 ?        S     10:38 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2467 ?        S     11:36 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2470 ?        S     12:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
16035 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
16036 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16037 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16038 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16039 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16040 ?        S<     0:00 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
16041 ?        S<     0:01 python2.1 /usr/lib/opcenter/ocwstart.pyc --fcgi --dae
21075 ?        S      6:40 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
26904 ?        S      4:51 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
26905 ?        S      4:28 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30035 ?        S      2:35 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30036 ?        S      2:48 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30037 ?        S      3:16 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30230 ?        S      2:38 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30231 ?        S      3:21 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30232 ?        S      2:48 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30618 ?        S      2:55 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
30620 ?        S      2:37 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31446 ?        S      2:16 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31448 ?        S      2:07 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31632 ?        S      2:22 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
31791 ?        S<     0:00 /usr/sbin/ocwhttpd -DSSL -d /usr/lib/opcenter -f fast
  431 ?        S      1:41 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  432 ?        S      1:37 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  434 ?        S      1:26 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  464 ?        S      1:33 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  465 ?        S      1:34 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
  467 ?        S      1:21 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 1506 ?        S      1:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2313 ?        S      0:41 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2315 ?        S      0:26 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2497 ?        S      0:45 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 2498 ?        S      0:44 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 3246 ?        S      0:26 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 3464 ?        RN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3661 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3662 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3663 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3664 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3665 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3666 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3667 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3668 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3669 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3670 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3671 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3672 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3673 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3674 ?        RN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3675 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3676 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3677 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3678 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3679 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3680 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3681 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3682 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3683 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3684 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3685 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3686 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3687 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3688 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3689 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3690 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3691 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3692 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3693 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3694 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3695 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3696 ?        SN     0:09 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3697 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3698 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3699 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3700 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3701 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3702 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3703 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3704 ?        RN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3705 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3706 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3707 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3708 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3709 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3710 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3711 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3712 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3713 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3714 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3715 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3716 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3717 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3718 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3719 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3720 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3721 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3722 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3723 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3724 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3725 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3726 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3727 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3728 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3729 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3730 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3731 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3732 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3733 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3734 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3735 ?        SN     0:09 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3736 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3737 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3738 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3739 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3741 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3748 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3749 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3750 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3751 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3760 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3761 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3762 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3763 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3764 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3765 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3766 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3767 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3768 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3769 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3773 ?        SN     0:11 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3774 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3775 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3777 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3780 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3781 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3782 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3787 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3788 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3789 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3798 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3799 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3800 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3801 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3802 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3803 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3804 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3805 ?        SN     0:08 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3806 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3807 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3808 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3809 ?        RN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3810 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3811 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3812 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3813 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3814 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3815 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3816 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3817 ?        SN     0:07 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3818 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3819 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3820 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3821 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3822 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3823 ?        SN     0:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3824 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3825 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3826 ?        SN     0:05 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3827 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3828 ?        SN     0:03 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3829 ?        SN     0:09 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 3934 ?        S      0:19 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4202 ?        S      0:02 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailSc
 4270 ?        S      0:17 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4435 ?        S      0:10 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4440 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4441 ?        S      0:08 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4442 ?        S      0:10 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4444 ?        S      0:07 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4445 ?        S      0:12 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4446 ?        S      0:17 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4468 ?        S      0:13 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4491 ?        S      0:10 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4493 ?        S      0:15 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4515 ?        S      0:12 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4517 ?        S      0:11 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4518 ?        S      0:13 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4598 ?        S      0:08 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4602 ?        S      0:11 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4609 ?        S      0:09 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4645 ?        S      0:03 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4747 ?        SN     0:06 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4748 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4749 ?        RN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4750 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4751 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4752 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4753 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4754 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4755 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4756 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4757 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4758 ?        SN     0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4759 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4862 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4863 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4879 ?        S      0:04 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4880 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4881 ?        S      0:07 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4882 ?        S      0:04 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4884 ?        S      0:05 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4909 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 4975 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4976 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4977 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4978 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4979 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4980 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4981 ?        SN     0:01 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4982 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4983 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4984 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 4985 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5017 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5018 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5019 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5020 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5021 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5022 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5023 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5024 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5025 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5026 ?        RN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5028 ?        SN     0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --bas
 5044 ?        S      0:03 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5136 ?        S      0:00 /usr/sbin/sshd
 5138 pts/1    S      0:00 -bash
 5192 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5193 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5194 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5195 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5196 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5197 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5198 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5199 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5200 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5201 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5202 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5203 ?        S      0:02 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5204 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5207 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5209 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5271 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5300 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5301 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5302 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5305 ?        S      0:01 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5309 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5319 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5320 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5322 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5324 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5325 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5327 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5328 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5329 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5330 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5331 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS
 5332 ?        S      0:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd_app.conf -DS


I really hope someone can help me out here.

THanks,
Derek


0
7thgencivic
Asked:
7thgencivic
  • 4
  • 3
1 Solution
 
gletiecqCommented:
A couple of comments...

You may want to segregate your database server and your http server.  Since the activity on each is related to the activity of the other, when one is clamoring for resources that's exactly the time when the other is as well.

You're running some apps that probably don't belong on that server, as it means you're opening virtually all of the ports on that box.  Sendmail, named, postgres, ftp, your python scripts, and everything else may better belong somewhere else.  If you want to secure a DB server or an http server, it's a lot easier when you lock down all the other stuff and move those other functions to another box.  Having everything on one box makes it hard to determine what can be locked down and hard to spot when you've got problems.

Google for async threading.  It looks like you're running multiple sync threads per process, and that doesn't scale too well above 500 or so users.  I think there's a recent article on osopinions.com about this where they look at this issue in WinXP, RH and Solaris.

Check out that the cache+free memory looks like when things are running smoothly versus when things are bogging down.  You've got a fair amount of memory on that box, so I don't suspect you're starving for more, but if you see a huge difference between these times it's probably worth throwing some more memory at the problem.

Hope this helps.

Greg

0
 
7thgencivicAuthor Commented:
I am running an ensim box, this is why all those are running. I have talked to ppl on forums.rackshack.net and ppl have the same box pretty much, or less and more users and it barely makes a dent.
On a max, I will get 250 concurrent users.

Derek
0
 
GnsCommented:
Your load average is kind of high (load average: 24.17, 23.87, 18.49) for a one CPU system, but you seem to have plenty of memory left (free+cached ~400MiB), so ... I would have guessed that you suffered from an IO bootleneck "somewhere".

One reason for high load averages is that processes in state D add 1 each to the average, but that should have shown up in the ps listing... hmmm.

I'd still look long and hard on the NIC driver and the disk subsystem. Some NIC drivers spend an inordinate time in kernel.

Gregs advice about dividing the workload on more machines is good (I'd go for at least 3... 1 web server, 1 database server and 1 for the rest).

-- Glenn
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
7thgencivicAuthor Commented:
Thanks Glenn, but that is not practical given my situation.
I may be able to do 2 boxes, but now more. THere has to be something that I am able to do, given the fact that other ppl have the same type of setup and have no issues like this?

Derek
0
 
GnsCommented:
Yes, start by looking at getting a "baseline" for normal operation with regards to memory status (vmstat, top sorted by (M)emory ...), processes, IO stats etc. the sar package should be useable, albeit a bit rudimentary:-).
When you have that, you will probably see where the problem areas are. Booring but effective.

Another thing that can drive the load average through the roof, without leaving much trace, is a fork-bomb or processes behaving in the same manner (spawning rather fast, but perhaps not living so long that you "register" them).
If you note a very rapid progression in process IDs, this might be it.

Say that some bozo (I had a chineese host attact our mail-server today...) really stresses out sendmail, or rather your mailfilter.... could be bad.

I recommend the swordfish book ("System Performance Tuning, 2nd edition" from Oreilly) if you are interrested in some in-depth thoughts on performance tuning.

-- Glenn
0
 
GnsCommented:
Is that ps listing from the same time as the top?

-- Glenn
0
 
7thgencivicAuthor Commented:
no it is not.
0
 
GnsCommented:
OK, then we cannot right off there being a lot of processes in state D (non-interruptible wait-state, which as you know is most often waiting for IO to complete in some form (often disk or network IO)).
When you have these exorbitant loads, do
ps auxww|grep " D "
or just page through the listing, and make a note on which/how many processes is in state D.
If there are a fair amount (20+) and they seem to be of the same "type" (mostly network (http, mail etc.) or mostly disk (mysql ... perhaps the mailfilter)), you've got a lead on where the (probable) bottleneck is.

Getting the "baseline" will make it easier to "see" the extraordinary conditions.

Good Luck
-- Glenn
0

Featured Post

Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now