Solved

server goes day at 4am everyday

Posted on 2006-11-05
17
421 Views
Last Modified: 2011-09-20
hello,
my server stops responding at 4am everyday so I have to manually reboot it!
my server is running on centos v4.4 can someone please help me?
I really need to fix this problem!
0
Comment
Question by:Sabrin
  • 5
  • 5
  • 4
  • +1
17 Comments
 
LVL 43

Expert Comment

by:ravenpl
ID: 17875849
> my server stops responding at 4am everyday so I have to manually reboot it!

Cat You verify what applications are beeing run from crontab daily?
/etc/crontab
and content from /etc/cron.daily/
0
 

Author Comment

by:Sabrin
ID: 17875955
[(06:39 AM)][(root@dedicated)] [(~)] $ vi /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

[(06:40 AM)][(root@dedicated)] [(~)] $ cd /etc/cron.daily/

[(06:36 AM)][(root@dedicated)] [(/etc/cron.daily)] $ ll
total 164
lrwxrwxrwx  1 root root   28 Oct 13 21:03 00-logwatch -> ../log.d/scripts/logwatch.pl
-rwxr-xr-x  1 root root  418 Apr 26  2006 00-makewhatis.cron
-rwxr-xr-x  1 root root  276 Feb 21  2005 0anacron
yum.cron (END)
-rw-r--r--  1 root root  851 Sep 28 14:13 0fpusage
-rwxr-xr-x  1 root root   85 Oct 16 15:17 0virtualhosting.site1
-rwxr-xr-x  1 root root   86 Oct 16 15:32 0virtualhosting.site10
-rwxr-xr-x  1 root root   86 Oct 16 15:33 0virtualhosting.site11
-rwxr-xr-x  1 root root   86 Oct 16 15:34 0virtualhosting.site12
-rwxr-xr-x  1 root root   86 Oct 16 15:34 0virtualhosting.site13
-rwxr-xr-x  1 root root   86 Oct 16 14:19 0virtualhosting.site14
-rwxr-xr-x  1 root root   86 Oct 17 14:13 0virtualhosting.site15
-rwxr-xr-x  1 root root   85 Oct 16 15:20 0virtualhosting.site2
-rwxr-xr-x  1 root root   85 Oct 16 15:24 0virtualhosting.site3
-rwxr-xr-x  1 root root   85 Oct 16 15:25 0virtualhosting.site4
-rwxr-xr-x  1 root root   85 Oct 16 15:28 0virtualhosting.site5
-rwxr-xr-x  1 root root   85 Oct 16 15:30 0virtualhosting.site6
-rwxr-xr-x  1 root root   85 Oct 16 15:30 0virtualhosting.site7
-rwxr-xr-x  1 root root   85 Oct 16 15:31 0virtualhosting.site8
-rwxr-xr-x  1 root root   85 Oct 16 15:32 0virtualhosting.site9
-rwxr-xr-x  1 root root  241 Sep 28 14:12 1gen_analog_data
-rwxr-x---  1 root root  168 Sep 28 14:12 1hup_apache_logs
-rw-r--r--  1 root root  797 Feb 21  2005 certwatch
-rwx------  1 root root  257 Sep 28 14:15 check_quotas
-rwxr-xr-x  1 root root  998 Jun  1  2004 clean.quarantine
-rwxr-xr-x  1 root root  456 Jan  9  2006 freshclam
-rwxr-xr-x  1 root root  180 Aug 22  2005 logrotate
-rwxr-xr-x  1 root root  345 Jul 17 08:01 poprelay.custodiat.pl
-rwxr-xr-x  1 root root 2133 Dec  1  2004 prelink
-rwx------  1 root root   83 Sep 28 14:10 rm_oneclick
-rwxr-xr-x  1 root root  104 Aug 12 17:01 rpm
-rwxr-xr-x  1 root root  121 Aug 21  2005 slocate.cron
-rwxr-xr-x  1 root root  286 Feb 21  2005 tmpwatch
-rwx------  1 root root   68 Sep 28 14:10 vacuumdb
-rwxr-xr-x  1 root root   51 Sep 28 14:13 wltouch.daily
-rwxr-xr-x  1 root root  158 Aug 12 13:40 yum.cron
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 17876212
One of those scripts from /etc/cron.daily  is downing the server. Simply try running them one by one - maybe one of them will not return ;)
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 17877087
Hi,

This might as well be as a part of a hardware failure sch as RAM or Hard Disk failure. And since the system llocks yp I'll suggest you to reboot in memetest mode gram the grub and check your memory throughly. (IT seems to me that it is very likely to be a RAM error since the system locks up.)

Then you can fsck all your volumes.
0
 

Author Comment

by:Sabrin
ID: 17878508
how can I test it ?
0
 
LVL 30

Expert Comment

by:Kerem ERSOY
ID: 17878615
- Just boot-off the install CD
- When theinitial prompt appears type "memtest86" enter
It will start the RAM test.
0
 

Author Comment

by:Sabrin
ID: 17879317
whats this script in /etc/cron.daily?

#!/bin/sh
/usr/sbin/chroot /home/virtual/site1/fst /usr/bin/run-parts /etc/cron.daily
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 17879329
It fires chained cron jobs from directory: /home/virtual/site1/fst/etc/cron.daily
Note - that they are run as root, therefore may cause damage as well.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:Sabrin
ID: 17879368
this are the scripts in that dir!
I think one of the sites are taking down the server!

[(03:38 AM)][(root@dedicated)] [(/home/virtual/site1/fst/etc/cron.daily)] $ ls

00-makewhatis.cron  00updatestats  rmattachments

[(03:38 AM)][(root@dedicated)] [(/home/virtual/site1/fst/etc/cron.daily)] $ less 00-makewhatis.cron

#!/bin/bash

LOCKFILE=/var/lock/makewhatis.lock

# the lockfile is not meant to be perfect, it's just in case the
# two makewhatis cron scripts get run close to each other to keep
# them from stepping on each other's toes.  The worst that will
# happen is that they will temporarily corrupt the database...
[ -f $LOCKFILE ] && exit 0
trap "{ rm -f $LOCKFILE ; exit 255; }" EXIT
touch $LOCKFILE
makewhatis -u -w
exit 0

[(03:39 AM)][(root@dedicated)] [(/home/virtual/site1/fst/etc/cron.daily)] $ less 00updatestats

#!/bin/sh
# --------------------------------------------------------------------------
# Copyright (c) Ensim Corporation 2000, 2001
# ENSIM CORPORATION - ALL RIGHTS RESERVED

# --------------------------------------------------------------------------
# $Id: updatestats,v 1.13 2005/07/28 10:47:00 mborate Exp $
# $Name: epl-10-1-0-branch $
# --------------------------------------------------------------------------
APACHE_LOG=/var/log/httpd/access_log
/usr/lib/opcenter/analog/updatestats.pl /var/usage/web $APACHE_LOG >/dev/null 2>/dev/null
/usr/lib/opcenter/analog/updatestats.pl /var/usage/ftp /var/log/xferlog  >/dev/null 2>/dev/null

[(03:39 AM)][(root@dedicated)] [(/home/virtual/site1/fst/etc/cron.daily)] $ less rmattachments

#!/bin/sh
# --------------------------------------------------------------------------
# Copyright (c) Ensim Corporation 2000, 2001
# ENSIM CORPORATION - ALL RIGHTS RESERVED

# --------------------------------------------------------------------------

find /var/www/squirrelmail-attachments/ -type f -mtime +1 -exec rm -rf {} \;
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 17879377
> I think one of the sites are taking down the server!
Possibly - You have to scan all sites for their crons. The scripts You provided are nothing suspicious, hence running many such scripts concurrently will cause very high load average, and in fact I saw servers that under high LA stop responding. This can be the cause.
0
 

Author Comment

by:Sabrin
ID: 17879663
is there a way to run each script from daily cron every 2 mins after each other ?
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 17879688
Cron does run it's jobs sequencially anyway.
But the job itself may go into background - then problem arises. Maybe the sites crons are run in parallel?
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 17885993
run-parts is a shell script, so you could indeed modify it to kick off a job every 2 minutes.
0
 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 100 total points
ID: 17886085
"locate run-parts" to see where it is. I guess you are hoping to use the time of failure to identify which script is causing the problem. You can also modify run-parts to use the logger command to keep you informed of what it's doing.
When you say the server stops responding, do you mean it stops responding to network requests or is there no response at all from the console (no Alt-F6 if running X, no console echo, reset button or power cycle needed to reboot).
Perhaps Linux ran out of swap? Bad nfs connections can also give these symptoms. I'm inclined to think the 4am failure is too consistent to be hardware, but you never know.
0
 
LVL 30

Accepted Solution

by:
Kerem ERSOY earned 400 total points
ID: 17886088
It is obvious that your system is locked during you run cron.daily.

- Did you try to check your mem ? It is easy just boot the install cd and and type "memtest86" (without " characters) and enter. It will get into test memory mode and it will start immediately. When you're doe just press ESC to reboot.

- Did you try to launch the scripts manually ? Does it lock the computer when launched this way ? You can loanch the script such as:
bash -x /script/to/run

so that it will print the command it runs interactively during the execution. You may also include the -x switch to the beginning of your files in the cron.daily folder like that

#!/bin/bash -x

in this case the utput fom your script will be emailed to you while running. But the preferred method is running them manually:
- Switchh yo the folder:
cd /etc/cron.daily
- Run all executable files in the directory one by one with bash -x
./00-makewhatis.cron
etc.
0
 
LVL 30

Assisted Solution

by:Kerem ERSOY
Kerem ERSOY earned 400 total points
ID: 17886187
You may also add echo commands in your code and redirect output to a logfile:

cd /var/log
touch cronts

Run-parts is only a front-end it simply runs all the executable files  in a directory.

cd /etc/cron.daily

edit files and add something like this:

echo $(date +'%b  %e %R:%S') $(basename $0)  CP01 >> /var/log/cronts

before each line don't forget to replace the CP01 with some meaningful text. Dont for get to wrap text in ".." if it will have more than one word.

0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Join & Write a Comment

If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now