cron doesn't work on one LINUX machine

i have 2 Linux boxes, RHEL
on both machine i set cron jobs to run as a specified user, but i noticed they are not working on one machine. this is the contents of the input file from cron, as you see it does not rely on any environment setting, and the command file to be executed has all permissions correctly.

25 13 * * *  /home/solr/admin/backup_solr_data.sh /solrdata /backup/shared/solrdata > /home/solr/admin/backup_solr_data.log 2>&1

the log file does not get created, the folder has 777 permission, and when i look at /var/log/cron i do not see that cron even knows about the script:

Apr 11 13:21:02 swsolr crontab[8811]: (root) END EDIT (root)
Apr 11 13:21:07 swsolr crontab[8813]: (root) LIST (root)
Apr 11 13:21:12 swsolr crontab[8814]: (root) BEGIN EDIT (root)
Apr 11 13:21:19 swsolr crontab[8814]: (root) REPLACE (root)
Apr 11 13:21:19 swsolr crontab[8814]: (root) END EDIT (root)
Apr 11 13:21:21 swsolr crontab[8816]: (root) LIST (root)
Apr 11 13:30:01 swsolr crond[8820]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Apr 11 13:40:01 swsolr crond[8822]: (root) CMD (/usr/lib64/sa/sa1 1 1)

but issuing "crontab -l" you see the entry:

25 13 * * *  /home/solr/admin/backup_solr_data.sh /solrdata /backup/shared/solrdata > /home/solr/admin/backup_solr_data.log 2>&1

on the other machine it's working, and i see the execution on /var/log/cron.

both machines do not have cron.allow, and an empty cron.deny

i tried the same cron for root and it's the same, so not a specific user issue.

crond is running ...
[root@swsolr admin]# ps ax | grep cron
 2575 ?        Ss     0:01 crond
 8853 pts/0    S+     0:00 grep cron
===========================================
any ideas ??
LVL 1
structuredwebAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

wesly_chenCommented:
1. check the cron is working or not
* * * * *  ls -l /tmp > /home/solr/admin/test.log 2>&1
And check whether /home/solr/admin/test.log is generated or not.

2. Do all the commands in /home/solr/admin/backup_solr_data.sh exist on server swsolr?
0
jgiordanoCommented:
do you use pam_access.so in your security configurations? if so you may need to modify /etc/security/access.conf
0
structuredwebAuthor Commented:
OK, tried that, it seems the basic cron mechanism doesn't work
===============

[root@swsolr admin]# crontab -l
* * * * * ls -l /tmp > /home/solr/admin/test.log 2>&1

[root@swsolr admin]# ls -la  /home/solr/admin/test.log
ls: /home/solr/admin/test.log: No such file or directory


[root@swsolr admin]# tail /var/log/cron
Apr 11 14:06:13 swsolr crontab[8838]: (root) LIST (root)
Apr 11 14:10:01 swsolr crond[8845]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Apr 11 14:12:01 swsolr crontab[8848]: (root) LIST (root)
Apr 11 14:14:49 swsolr crontab[8850]: (root) LIST (root)
Apr 11 14:20:01 swsolr crond[8855]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Apr 11 14:30:01 swsolr crond[8858]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Apr 11 14:40:01 swsolr crond[8860]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Apr 11 14:45:11 swsolr crontab[8886]: (root) REPLACE (root)
Apr 11 14:45:14 swsolr crontab[8887]: (root) LIST (root)
Apr 11 14:46:00 swsolr crontab[8891]: (root) LIST (root)
[root@swsolr admin]#

2. Do all the commands in /home/solr/admin/backup_solr_data.sh exist on server swsolr?

they do, but i think we should agree this is more basic issue


0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

structuredwebAuthor Commented:

jgiordano:
do you use pam_access.so in your security configurations? if so you may need to modify /etc/security/access.conf
========================

i don;t even know what it is ... /etc/security/access.conf has only comment lines
0
jgiordanoCommented:
I missed that you ran it as root and it still did not work, access.conf maybe an issue if NO crons are running and you have it configured.

run on both machines - grep pam_access.so /etc/pam.d/*


what about writing out to /tmp, maybe it is with the dest

* * * * * ls -l /tmp > /tmp/test.log 2>&1



0
jgiordanoCommented:
here is a good explanation of the cron issues with acccess.conf

http://lazysystemadmin.blogspot.com/2010/09/enable-cron-for-locked-account-in-linux.html

In some environments, there are times when locked application/databases accounts need to run some cron jobs. In linux, by default, a locked account can not run the cron job.

We can enable this  by editing a specific setting is disabled in /etc/pam.d/crond file.

Here is the details:


# cat /etc/pam.d/crond


#
# The PAM configuration file for the cron daemon
#
auth       sufficient pam_rootok.so
auth       required   pam_stack.so service=system-auth
auth       required   pam_env.so
account    required   pam_stack.so service=system-auth
# account    required   pam_access.so
session    required   pam_limits.so
session    required   pam_loginuid.so

This example is working in Redhat Linux. In the /etc/pam.d/crond file, if we disable "account required pam_access.so" line the cron started working again for the locked account as well.
0
structuredwebAuthor Commented:
yes, it's not the PAM config. i commented the same line, restarted crond, and the same issue... and it;s happening to root as well. on the "good" machine that line was there, uncommented.

i also tried a cron job writing to /tmp, same problem ... /var/log/cron doesn't show that it even tried


[root@swsolr tmp]# crontab -l
36 15 * * * ls -l / > /tmp/test.log 2>&1
[root@swsolr tmp]#
0
jgiordanoCommented:
can you fully qualify the ls -l cronjob and put it in roots cron to see if it runs.
0
structuredwebAuthor Commented:

can you fully qualify the ls -l cronjob and put it in roots cron to see if it runs.

not sure what you mean by "fully qualify". i did create the cron entry as root for the ls -l /tmp command, and it wasn't executed. if you recall, /var/log/cron did not show an execution of this item at all.

i just created this for root:

52 10 * * * ls -l /tmp > /tmp/test.log 2>&1

waited until 10:54, nothing ... no trace of output file, no trace of the execution in /var/log/cron
0
jgiordanoCommented:
/bin/ls /usr/bin/ls (whichever is appropriate)
0
jgiordanoCommented:
Do you have SE linux enabled?
0
structuredwebAuthor Commented:
UPDATE, ITS WORKING NOW !!!

i looked in /var/con/secure , honestly i just looked at all log files in /var/log as i was at my wits end, and i saw these entries:

Apr 12 14:20:01 swsolr crond[9990]: pam_unix(crond:account): account root has password changed in future

i looked it up, and someone mentioned check the date, so i realized the system date was 2001 instead of 2011!

changed, it and immediately cron started to work... i just tried for the "solr" user, original job, no problem...


========================================================================

i tried the /bin/ls, to no avail. i think this really more fundamental, as /var/log/cron proves the system does not even try to run whatever is listed as the cron job for this user, including root.

the 2 machines, the one which works and the other which does not, were built using the same Linux. SE is not enabled unless it was without my knowledge, how would i know that?

0
structuredwebAuthor Commented:
typo above: /var/log/secure
0
wesly_chenCommented:
> system date was 2001 instead of 2011
Woo, you might want to setup ntpd to sync the time
http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch24_:_The_NTP_Server
0
structuredwebAuthor Commented:
i sure learned my lesson.

still, you would think Linux would have friendlier/more accurate error messages but...

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
structuredwebAuthor Commented:
in case of cron jobs not running at all - check the system date and the /var/log/secure for crond messages
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.