asked on

cron ignoring specified time

I am running Slackware 64 14.1 kernel 3.10.17. A couple of days ago cron started acting funny. For the daily backup job `crontab -l` shows:

$ crontab -l | grep /backup
30 1 * * tue-sat /bin/bash -c "/root/backup >>/root/logfiles/backup.log 2>&1"

Open in new window

Clearly, it shows running at 1:30AM. Yet starting on Wednesday it began running at 3:30AM (log file):

BACKUP STARTED Wed Jul  1 01:30:02 EDT 2015
BACKUP STARTED Thu Jul  2 01:30:01 EDT 2015
BACKUP STARTED Fri Jul  3 01:30:01 EDT 2015
BACKUP STARTED Sat Jul  4 01:30:01 EDT 2015
BACKUP STARTED Tue Jul  7 01:30:01 EDT 2015
BACKUP STARTED Wed Jul  8 01:30:01 EDT 2015
BACKUP STARTED Thu Jul  9 01:30:01 EDT 2015
BACKUP STARTED Fri Jul 10 01:30:01 EDT 2015
BACKUP STARTED Sat Jul 11 01:30:01 EDT 2015
BACKUP STARTED Tue Jul 14 01:30:01 EDT 2015
BACKUP STARTED Wed Jul 15 03:30:01 EDT 2015
BACKUP STARTED Thu Jul 16 03:30:01 EDT 2015
BACKUP STARTED Fri Jul 17 03:30:01 EDT 2015

Open in new window

Friday (today), after 1:30AM I deleted and reloaded the crontab keeping the 1:30 time, but adding a line to run at 1:55. It did not run at 1:55. I removed the 1:55 line and reloaded crontab, but it still ran at 3:30. Note that nothing is in the /var/log/cron file indicating an error running at 1:30 (or 1:55).

This is crazy. What's up? How do I "reset" cron?

More info: I was restoring some files to an nfs mounted drive through the wee-hours of Wednesday morning and I did change the crontab entry for the the backup job, pushing it back first to 2:30, then 3:30, 4:30, etc to 6:30, then finally restoring it back to 1:30. That's the only mucking-about I did, but perhaps somehow the change to 3:30 "stuck" and now I can't change it?

Zephyr ICT

You probably checked this already but did your timezone possibly change?

You could also check the hwclock ( hwclock --systohc)

arnold

Along the path Spravtek double check the localtime on which cron runs versus the backup script.

Compare the log above to the /var/log/cron or /var/cron/

ls -l /etc/localtime
find /usr/share/zoneinfo -size <size of the localtime above>
i.e. your system is in one timezone, but your script sets EDT as the TZ

meaning your system Time is US/Mountain while your backup script uses TZ as "US/Eastern"

login into the system and run date
what is the output? does it match the correct time for the reported timezone?

SOLUTION

Gerwin Jansen

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Mark

ASKER

spravtek: Timezone still EDT. Hardware clock same as system clock:

$ hwclock -r
Fri 17 Jul 2015 04:21:06 PM EDT -0.531770 seconds
1 16:21:06 root@webserver:~/

Clock synchronized by ntp, looks good:

> ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.0.2     173.44.32.10     3 u 1076 1024  377    0.178   -0.243   0.189
+97.107.129.217  200.98.196.212   2 u  708 1024  337   30.206    4.005   0.941
 127.127.1.0     .LOCL.          10 l  75h   64    0    0.000    0.000   0.000

Open in new window

arnold: Nothing in /var/log/cron related to this job

$ ls -l /etc/localtime
-rw-r--r-- 1 root root 3519 2015-06-04 15:57 /etc/localtime

$ find /usr/share/zoneinfo -size 3519c -exec ls -l \{\} \;
-rw-r--r-- 3 root root 3519 Sep 15  2013 /usr/share/zoneinfo/posixrules
-rw-r--r-- 3 root root 3519 Sep 15  2013 /usr/share/zoneinfo/America/New_York
-rw-r--r-- 3 root root 3519 Sep 15  2013 /usr/share/zoneinfo/US/Eastern
-rw-r--r-- 2 root root 3519 Sep 15  2013 /usr/share/zoneinfo/posix/America/New_York
-rw-r--r-- 2 root root 3519 Sep 15  2013 /usr/share/zoneinfo/posix/US/Eastern

$ date
Fri Jul 17 16:30:03 EDT 2015            # Matches time and zone

Open in new window

Looks like the timezone is set correctly.

Gerwin Jansen:

What is the content of that backup script, the part that is creating the log entry?

#!/bin/bash

export notify=obfuscated@ohprs.org

echo -e "\n==================================="
echo -n "BACKUP STARTED "
start=`date`
echo "$start"

Open in new window

I think there is something wrong with cron for this user (root). Other crons for other users are running at the correct time.

Mark

ASKER

OK, it's official. I cannot change cron for any user. I've added a cron entry for root and done the same for 2 other users, one of which did not have a crontab to begin with. None of the new entries ran. Whatever crons were set as of early Wednesday morning are still running, but I can't change, remove or add cron entries. Doing `crontab -l` shows my changes, but is not what runs.

cron is messed up. How do I reset fix?

SOLUTION

arnold

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Mark

ASKER

Arnold:

Try adding this this,
29 1 * * tue-sat /bin/echo Cron task `/bin/date` >> /root/logfiles/backup.log

Tried something similar:

* * * * * /usr/bin/date >>/root/testme.log

nothing in testme.log. As I mentioned in my last post, no new cron entries are being run for any user.

While your localtime is set to US/EASTERN, your cron is running a different time zone.

Don't think so. cron jobs that were in the crontab for this and other users as of between 2:00AM and 3:30AM Wednesday continue to run and do so at the specified system time.

> ps -ef | grep crond | grep -v grep
root      1003     1  0 Jul14 ?        00:00:03 /usr/sbin/crond -l notice -m sysadmin

Open in new window

Not sure what the date is, but interesting that it is Jul14, before I noticed this problem.

/var/spool/cron/crontabs/cron.update has 12 entries for root, and several others for the other users I've tested. Clearly, crond is not updating changed crontabs.

Can't reboot at the moment. Will try/stopping restarting crond. Will post back results.

ASKER CERTIFIED SOLUTION

Zephyr ICT

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Mark

ASKER

Well, stopping and restarting crond did the trick. The 1:30AM job is back running at 1:30 and new jobs work. I can't believe I didn't think to do the restart to begin with. I don't know what caused it to mess up in the first place.

Zephyr ICT

Normally that would be the first thing I ask, but we're all professionals here aren't we ;-)

Don't worry, we've all been there, looking for a solution while it was staring us in the face.
Thanks for getting back at us!

Mark

ASKER

Thanks all. Duh!