cron ignoring specified time

I am running Slackware 64 14.1 kernel 3.10.17. A couple of days ago cron started acting funny. For the daily backup job `crontab -l` shows:
$ crontab -l | grep /backup
30 1 * * tue-sat /bin/bash -c "/root/backup >>/root/logfiles/backup.log 2>&1"

Open in new window

Clearly, it shows running at 1:30AM. Yet starting on Wednesday it began running at 3:30AM (log file):
BACKUP STARTED Wed Jul  1 01:30:02 EDT 2015
BACKUP STARTED Thu Jul  2 01:30:01 EDT 2015
BACKUP STARTED Fri Jul  3 01:30:01 EDT 2015
BACKUP STARTED Sat Jul  4 01:30:01 EDT 2015
BACKUP STARTED Tue Jul  7 01:30:01 EDT 2015
BACKUP STARTED Wed Jul  8 01:30:01 EDT 2015
BACKUP STARTED Thu Jul  9 01:30:01 EDT 2015
BACKUP STARTED Fri Jul 10 01:30:01 EDT 2015
BACKUP STARTED Sat Jul 11 01:30:01 EDT 2015
BACKUP STARTED Tue Jul 14 01:30:01 EDT 2015
BACKUP STARTED Wed Jul 15 03:30:01 EDT 2015
BACKUP STARTED Thu Jul 16 03:30:01 EDT 2015
BACKUP STARTED Fri Jul 17 03:30:01 EDT 2015

Open in new window

Friday (today), after 1:30AM I deleted and reloaded the crontab keeping the 1:30 time, but adding a line to run at 1:55. It did not run at 1:55. I removed the 1:55 line and reloaded crontab, but it still ran at 3:30. Note that nothing is in the /var/log/cron file indicating an error running at 1:30 (or 1:55).

This is crazy. What's up? How do I "reset" cron?

More info: I was restoring some files to an nfs mounted drive through the wee-hours of Wednesday morning and I did change the crontab entry for the the backup job, pushing it back first to 2:30, then 3:30, 4:30, etc to 6:30, then finally restoring it back to 1:30. That's the only mucking-about I did, but perhaps somehow the change to 3:30 "stuck" and now I can't change it?
LVL 1
MarkAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Zephyr ICTCloud ArchitectCommented:
You probably checked this already but did your timezone possibly change?

You could also check the hwclock ( hwclock --systohc)
arnoldCommented:
Along the path Spravtek double check the localtime on which cron runs versus the backup script.

Compare the log above to the /var/log/cron or /var/cron/

ls -l /etc/localtime
find /usr/share/zoneinfo -size <size of the localtime above>
i.e. your system is in one  timezone, but your script sets EDT as the TZ

meaning your system Time is US/Mountain while your backup script uses TZ as "US/Eastern"


login into the system and run date
what is the output? does it match the correct time for the reported timezone?
Gerwin Jansen, EE MVETopic Advisor Commented:
What is the content of that backup script, the part that is creating the log entry?

You're of course not traveling with a notebook and crossing timezones I presume? ;)
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

MarkAuthor Commented:
spravtek: Timezone still EDT. Hardware clock same as system clock:

$ hwclock -r
Fri 17 Jul 2015 04:21:06 PM EDT  -0.531770 seconds
1 16:21:06 root@webserver:~/

Clock synchronized by ntp, looks good:
> ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.0.2     173.44.32.10     3 u 1076 1024  377    0.178   -0.243   0.189
+97.107.129.217  200.98.196.212   2 u  708 1024  337   30.206    4.005   0.941
 127.127.1.0     .LOCL.          10 l  75h   64    0    0.000    0.000   0.000

Open in new window

arnold: Nothing in /var/log/cron related to this job
$ ls -l /etc/localtime
-rw-r--r-- 1 root root 3519 2015-06-04 15:57 /etc/localtime

$ find /usr/share/zoneinfo -size 3519c -exec ls -l \{\} \;
-rw-r--r-- 3 root root 3519 Sep 15  2013 /usr/share/zoneinfo/posixrules
-rw-r--r-- 3 root root 3519 Sep 15  2013 /usr/share/zoneinfo/America/New_York
-rw-r--r-- 3 root root 3519 Sep 15  2013 /usr/share/zoneinfo/US/Eastern
-rw-r--r-- 2 root root 3519 Sep 15  2013 /usr/share/zoneinfo/posix/America/New_York
-rw-r--r-- 2 root root 3519 Sep 15  2013 /usr/share/zoneinfo/posix/US/Eastern

$ date
Fri Jul 17 16:30:03 EDT 2015            # Matches time and zone

Open in new window

Looks like the timezone is set correctly.

Gerwin Jansen:
What is the content of that backup script, the part that is creating the log entry?
#!/bin/bash

export notify=obfuscated@ohprs.org

echo -e "\n==================================="
echo -n "BACKUP STARTED "
start=`date`
echo "$start"

Open in new window

I think there is something wrong with cron for this user (root). Other crons for other users are running at the correct time.
MarkAuthor Commented:
OK, it's official. I cannot change cron for any user. I've added a cron entry for root and done the same for 2 other users, one of which did not have a crontab to begin with. None of the new entries ran. Whatever crons were set as of early Wednesday morning are still running, but I can't change, remove or add cron entries. Doing `crontab -l` shows my changes, but is not what runs.

cron is messed up. How do I reset fix?
arnoldCommented:
Try adding this this,
29 1 * * tue-sat /bin/echo Cron task `/bin/date` >> /root/logfiles/backup.log

See what you have in the log.

While your localtime is set to US/EASTERN, your cron is running a different time zone.

/var/log/cron there has to be an entry there for 1:30

ps -ef | grep crond

Without cron start job including a TZ that sets the timezone to another one, the localtime settings should match.
The shell is set separately or can be.
MarkAuthor Commented:
Arnold:
Try adding this this,
 29 1 * * tue-sat /bin/echo Cron task `/bin/date` >> /root/logfiles/backup.log
Tried something similar:

* * * * * /usr/bin/date >>/root/testme.log

nothing in testme.log. As I mentioned in my last post, no new cron entries are being run for any user.

While your localtime is set to US/EASTERN, your cron is running a different time zone.
Don't think so. cron jobs that were in the crontab for this and other users as of between 2:00AM and 3:30AM Wednesday continue to run and do so at the specified system time.
> ps -ef | grep crond | grep -v grep
root      1003     1  0 Jul14 ?        00:00:03 /usr/sbin/crond -l notice -m sysadmin

Open in new window

Not sure what the date is, but interesting that it is Jul14, before I noticed this problem.

/var/spool/cron/crontabs/cron.update has 12 entries for root, and several others for the other users I've tested. Clearly, crond is not updating changed crontabs.

Can't reboot at the moment. Will try/stopping restarting crond. Will post back results.
Zephyr ICTCloud ArchitectCommented:
Cron hasn't been restarted? That could help :-)

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
MarkAuthor Commented:
Well, stopping and restarting crond  did the trick. The 1:30AM job is back running at 1:30 and new jobs work. I can't believe I didn't think to do the restart to begin with. I don't know what caused it to mess up in the first place.
Zephyr ICTCloud ArchitectCommented:
Normally that would be the first thing I ask, but we're all professionals here aren't we ;-)

Don't worry, we've all been there, looking for a solution while it was staring us in the face.
Thanks for getting back at us!
MarkAuthor Commented:
Thanks all. Duh!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.