Sabrin
asked on
server goes day at 4am everyday
hello,
my server stops responding at 4am everyday so I have to manually reboot it!
my server is running on centos v4.4 can someone please help me?
I really need to fix this problem!
my server stops responding at 4am everyday so I have to manually reboot it!
my server is running on centos v4.4 can someone please help me?
I really need to fix this problem!
ASKER
[(06:39 AM)][(root@dedicated)] [(~)] $ vi /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin: /usr/bin
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
[(06:40 AM)][(root@dedicated)] [(~)] $ cd /etc/cron.daily/
[(06:36 AM)][(root@dedicated)] [(/etc/cron.daily)] $ ll
total 164
lrwxrwxrwx 1 root root 28 Oct 13 21:03 00-logwatch -> ../log.d/scripts/logwatch. pl
-rwxr-xr-x 1 root root 418 Apr 26 2006 00-makewhatis.cron
-rwxr-xr-x 1 root root 276 Feb 21 2005 0anacron
yum.cron (END)
-rw-r--r-- 1 root root 851 Sep 28 14:13 0fpusage
-rwxr-xr-x 1 root root 85 Oct 16 15:17 0virtualhosting.site1
-rwxr-xr-x 1 root root 86 Oct 16 15:32 0virtualhosting.site10
-rwxr-xr-x 1 root root 86 Oct 16 15:33 0virtualhosting.site11
-rwxr-xr-x 1 root root 86 Oct 16 15:34 0virtualhosting.site12
-rwxr-xr-x 1 root root 86 Oct 16 15:34 0virtualhosting.site13
-rwxr-xr-x 1 root root 86 Oct 16 14:19 0virtualhosting.site14
-rwxr-xr-x 1 root root 86 Oct 17 14:13 0virtualhosting.site15
-rwxr-xr-x 1 root root 85 Oct 16 15:20 0virtualhosting.site2
-rwxr-xr-x 1 root root 85 Oct 16 15:24 0virtualhosting.site3
-rwxr-xr-x 1 root root 85 Oct 16 15:25 0virtualhosting.site4
-rwxr-xr-x 1 root root 85 Oct 16 15:28 0virtualhosting.site5
-rwxr-xr-x 1 root root 85 Oct 16 15:30 0virtualhosting.site6
-rwxr-xr-x 1 root root 85 Oct 16 15:30 0virtualhosting.site7
-rwxr-xr-x 1 root root 85 Oct 16 15:31 0virtualhosting.site8
-rwxr-xr-x 1 root root 85 Oct 16 15:32 0virtualhosting.site9
-rwxr-xr-x 1 root root 241 Sep 28 14:12 1gen_analog_data
-rwxr-x--- 1 root root 168 Sep 28 14:12 1hup_apache_logs
-rw-r--r-- 1 root root 797 Feb 21 2005 certwatch
-rwx------ 1 root root 257 Sep 28 14:15 check_quotas
-rwxr-xr-x 1 root root 998 Jun 1 2004 clean.quarantine
-rwxr-xr-x 1 root root 456 Jan 9 2006 freshclam
-rwxr-xr-x 1 root root 180 Aug 22 2005 logrotate
-rwxr-xr-x 1 root root 345 Jul 17 08:01 poprelay.custodiat.pl
-rwxr-xr-x 1 root root 2133 Dec 1 2004 prelink
-rwx------ 1 root root 83 Sep 28 14:10 rm_oneclick
-rwxr-xr-x 1 root root 104 Aug 12 17:01 rpm
-rwxr-xr-x 1 root root 121 Aug 21 2005 slocate.cron
-rwxr-xr-x 1 root root 286 Feb 21 2005 tmpwatch
-rwx------ 1 root root 68 Sep 28 14:10 vacuumdb
-rwxr-xr-x 1 root root 51 Sep 28 14:13 wltouch.daily
-rwxr-xr-x 1 root root 158 Aug 12 13:40 yum.cron
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
[(06:40 AM)][(root@dedicated)] [(~)] $ cd /etc/cron.daily/
[(06:36 AM)][(root@dedicated)] [(/etc/cron.daily)] $ ll
total 164
lrwxrwxrwx 1 root root 28 Oct 13 21:03 00-logwatch -> ../log.d/scripts/logwatch.
-rwxr-xr-x 1 root root 418 Apr 26 2006 00-makewhatis.cron
-rwxr-xr-x 1 root root 276 Feb 21 2005 0anacron
yum.cron (END)
-rw-r--r-- 1 root root 851 Sep 28 14:13 0fpusage
-rwxr-xr-x 1 root root 85 Oct 16 15:17 0virtualhosting.site1
-rwxr-xr-x 1 root root 86 Oct 16 15:32 0virtualhosting.site10
-rwxr-xr-x 1 root root 86 Oct 16 15:33 0virtualhosting.site11
-rwxr-xr-x 1 root root 86 Oct 16 15:34 0virtualhosting.site12
-rwxr-xr-x 1 root root 86 Oct 16 15:34 0virtualhosting.site13
-rwxr-xr-x 1 root root 86 Oct 16 14:19 0virtualhosting.site14
-rwxr-xr-x 1 root root 86 Oct 17 14:13 0virtualhosting.site15
-rwxr-xr-x 1 root root 85 Oct 16 15:20 0virtualhosting.site2
-rwxr-xr-x 1 root root 85 Oct 16 15:24 0virtualhosting.site3
-rwxr-xr-x 1 root root 85 Oct 16 15:25 0virtualhosting.site4
-rwxr-xr-x 1 root root 85 Oct 16 15:28 0virtualhosting.site5
-rwxr-xr-x 1 root root 85 Oct 16 15:30 0virtualhosting.site6
-rwxr-xr-x 1 root root 85 Oct 16 15:30 0virtualhosting.site7
-rwxr-xr-x 1 root root 85 Oct 16 15:31 0virtualhosting.site8
-rwxr-xr-x 1 root root 85 Oct 16 15:32 0virtualhosting.site9
-rwxr-xr-x 1 root root 241 Sep 28 14:12 1gen_analog_data
-rwxr-x--- 1 root root 168 Sep 28 14:12 1hup_apache_logs
-rw-r--r-- 1 root root 797 Feb 21 2005 certwatch
-rwx------ 1 root root 257 Sep 28 14:15 check_quotas
-rwxr-xr-x 1 root root 998 Jun 1 2004 clean.quarantine
-rwxr-xr-x 1 root root 456 Jan 9 2006 freshclam
-rwxr-xr-x 1 root root 180 Aug 22 2005 logrotate
-rwxr-xr-x 1 root root 345 Jul 17 08:01 poprelay.custodiat.pl
-rwxr-xr-x 1 root root 2133 Dec 1 2004 prelink
-rwx------ 1 root root 83 Sep 28 14:10 rm_oneclick
-rwxr-xr-x 1 root root 104 Aug 12 17:01 rpm
-rwxr-xr-x 1 root root 121 Aug 21 2005 slocate.cron
-rwxr-xr-x 1 root root 286 Feb 21 2005 tmpwatch
-rwx------ 1 root root 68 Sep 28 14:10 vacuumdb
-rwxr-xr-x 1 root root 51 Sep 28 14:13 wltouch.daily
-rwxr-xr-x 1 root root 158 Aug 12 13:40 yum.cron
One of those scripts from /etc/cron.daily is downing the server. Simply try running them one by one - maybe one of them will not return ;)
Hi,
This might as well be as a part of a hardware failure sch as RAM or Hard Disk failure. And since the system llocks yp I'll suggest you to reboot in memetest mode gram the grub and check your memory throughly. (IT seems to me that it is very likely to be a RAM error since the system locks up.)
Then you can fsck all your volumes.
This might as well be as a part of a hardware failure sch as RAM or Hard Disk failure. And since the system llocks yp I'll suggest you to reboot in memetest mode gram the grub and check your memory throughly. (IT seems to me that it is very likely to be a RAM error since the system locks up.)
Then you can fsck all your volumes.
ASKER
how can I test it ?
- Just boot-off the install CD
- When theinitial prompt appears type "memtest86" enter
It will start the RAM test.
- When theinitial prompt appears type "memtest86" enter
It will start the RAM test.
ASKER
whats this script in /etc/cron.daily?
#!/bin/sh
/usr/sbin/chroot /home/virtual/site1/fst /usr/bin/run-parts /etc/cron.daily
#!/bin/sh
/usr/sbin/chroot /home/virtual/site1/fst /usr/bin/run-parts /etc/cron.daily
It fires chained cron jobs from directory: /home/virtual/site1/fst/et c/cron.dai ly
Note - that they are run as root, therefore may cause damage as well.
Note - that they are run as root, therefore may cause damage as well.
ASKER
this are the scripts in that dir!
I think one of the sites are taking down the server!
[(03:38 AM)][(root@dedicated)] [(/home/virtual/site1/fst/ etc/cron.d aily)] $ ls
00-makewhatis.cron 00updatestats rmattachments
[(03:38 AM)][(root@dedicated)] [(/home/virtual/site1/fst/ etc/cron.d aily)] $ less 00-makewhatis.cron
#!/bin/bash
LOCKFILE=/var/lock/makewha tis.lock
# the lockfile is not meant to be perfect, it's just in case the
# two makewhatis cron scripts get run close to each other to keep
# them from stepping on each other's toes. The worst that will
# happen is that they will temporarily corrupt the database...
[ -f $LOCKFILE ] && exit 0
trap "{ rm -f $LOCKFILE ; exit 255; }" EXIT
touch $LOCKFILE
makewhatis -u -w
exit 0
[(03:39 AM)][(root@dedicated)] [(/home/virtual/site1/fst/ etc/cron.d aily)] $ less 00updatestats
#!/bin/sh
# -------------------------- ---------- ---------- ---------- ---------- --------
# Copyright (c) Ensim Corporation 2000, 2001
# ENSIM CORPORATION - ALL RIGHTS RESERVED
# -------------------------- ---------- ---------- ---------- ---------- --------
# $Id: updatestats,v 1.13 2005/07/28 10:47:00 mborate Exp $
# $Name: epl-10-1-0-branch $
# -------------------------- ---------- ---------- ---------- ---------- --------
APACHE_LOG=/var/log/httpd/ access_log
/usr/lib/opcenter/analog/u pdatestats .pl /var/usage/web $APACHE_LOG >/dev/null 2>/dev/null
/usr/lib/opcenter/analog/u pdatestats .pl /var/usage/ftp /var/log/xferlog >/dev/null 2>/dev/null
[(03:39 AM)][(root@dedicated)] [(/home/virtual/site1/fst/ etc/cron.d aily)] $ less rmattachments
#!/bin/sh
# -------------------------- ---------- ---------- ---------- ---------- --------
# Copyright (c) Ensim Corporation 2000, 2001
# ENSIM CORPORATION - ALL RIGHTS RESERVED
# -------------------------- ---------- ---------- ---------- ---------- --------
find /var/www/squirrelmail-atta chments/ -type f -mtime +1 -exec rm -rf {} \;
I think one of the sites are taking down the server!
[(03:38 AM)][(root@dedicated)] [(/home/virtual/site1/fst/
00-makewhatis.cron 00updatestats rmattachments
[(03:38 AM)][(root@dedicated)] [(/home/virtual/site1/fst/
#!/bin/bash
LOCKFILE=/var/lock/makewha
# the lockfile is not meant to be perfect, it's just in case the
# two makewhatis cron scripts get run close to each other to keep
# them from stepping on each other's toes. The worst that will
# happen is that they will temporarily corrupt the database...
[ -f $LOCKFILE ] && exit 0
trap "{ rm -f $LOCKFILE ; exit 255; }" EXIT
touch $LOCKFILE
makewhatis -u -w
exit 0
[(03:39 AM)][(root@dedicated)] [(/home/virtual/site1/fst/
#!/bin/sh
# --------------------------
# Copyright (c) Ensim Corporation 2000, 2001
# ENSIM CORPORATION - ALL RIGHTS RESERVED
# --------------------------
# $Id: updatestats,v 1.13 2005/07/28 10:47:00 mborate Exp $
# $Name: epl-10-1-0-branch $
# --------------------------
APACHE_LOG=/var/log/httpd/
/usr/lib/opcenter/analog/u
/usr/lib/opcenter/analog/u
[(03:39 AM)][(root@dedicated)] [(/home/virtual/site1/fst/
#!/bin/sh
# --------------------------
# Copyright (c) Ensim Corporation 2000, 2001
# ENSIM CORPORATION - ALL RIGHTS RESERVED
# --------------------------
find /var/www/squirrelmail-atta
> I think one of the sites are taking down the server!
Possibly - You have to scan all sites for their crons. The scripts You provided are nothing suspicious, hence running many such scripts concurrently will cause very high load average, and in fact I saw servers that under high LA stop responding. This can be the cause.
Possibly - You have to scan all sites for their crons. The scripts You provided are nothing suspicious, hence running many such scripts concurrently will cause very high load average, and in fact I saw servers that under high LA stop responding. This can be the cause.
ASKER
is there a way to run each script from daily cron every 2 mins after each other ?
Cron does run it's jobs sequencially anyway.
But the job itself may go into background - then problem arises. Maybe the sites crons are run in parallel?
But the job itself may go into background - then problem arises. Maybe the sites crons are run in parallel?
run-parts is a shell script, so you could indeed modify it to kick off a job every 2 minutes.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Cat You verify what applications are beeing run from crontab daily?
/etc/crontab
and content from /etc/cron.daily/