?
Solved

Script Failing When Run From Cron - Runs Fine Manually

Posted on 2005-04-21
9
Medium Priority
?
721 Views
Last Modified: 2013-12-15
All right, I have a bizarre one here.  I have a simple backup shell script that I use in a number of places.  It backups up mysql using the mysql-hotcopy perl script, then tars the files and rsyncs the tar file to another server.  It runs fine from cron on various Redhat Linux builds, including Fedora Core 3 and Redhat 9.  It's failing on a "Red Hat Enterprise Linux ES release 3 (Taroon Update 4)" server with a gig of RAM, dual Xeon procs and all of the current Up2date updates.  It fails at the same place every time.  I have included the script below with a comment over the line that fails.  The comment says: "#FAILS HERE WHEN RUN FROM CRON"  It fails when it is supposed to change the ownership of the tar file with chown.  It's supposed to change the ownership to a lesser privledged user "backupguy" and then rsync the tar file to another server using that lower privledged user's SSH key.  

Here are some FACTS:

1) The script runs fine if executed manually (as in it succeeds in creating the tar file, changing ownership and rsyncing it to another machine)
2) The tar file created is 1.7  GBs, well with in an acceptable range for a tar file size.
3) I see no "chown" errors in /var/log/messages or in the log file the script is using /var/log/mysql-hotcopy.log
4) The script works from cron if I comment out the line right below the comment that says "# IF I COMMENT THE DB BACKUP BELOW THE SCRIPT WORKS".  This is the largest DB, about 3 GB uncompressed.  However, the strange thing is that the script does not fail on copying the large DB.  It copies it fine. Then the script continues and records success in the log file (see log file excerpt below): It then moves on and tars the file.  It's only once it gets to "chown" that it fails.
# Log file exerpt
 (Copying 34 files...
Copying indices for 0 files...
Unlocked tables.
mysqlhotcopy copied 11 tables (34 files) in 396 seconds (396 seconds overall).
# End log file excerpt
5) I have tried running it with nohup, to no avail.
6) The cron job runs under root's crontab.
7) I thought that maybe I need to change back to /root to run the chown command, but that didn't help either and remember it runs the "chown" command properly if I don't back up the big DB.
8) I have also tried chown with "backupguy.backupguy" instead of the colon.

My guesses as to what the problem might be:
1) Cron has a timeout that I'm not aware of
2) The chown command is failing silently.  I will need to find a way to capture that, since the log file captures nothing.
3) The script is running out of memory.
4) I really have no idea. :)  Any help is tremendously appreciated.


The script is below:

#!/bin/bash
#
# Script to backup all Mysql DBs
#
# Removing slq files
#
/bin/rm -rf /backup/mysql/daily/*
#
# Log start time
#
echo 'START TIME: ' >> /var/log/mysql-hotcopy.log
date >> /var/log/mysql-hotcopy.log
echo 'Copying DBs' >> /var/log/mysql-hotcopy.log
#
# IF I COMMENT THE DB BACKUP BELOW THE SCRIPT WORKS
/usr/local/mysql/bin/mysqlhotcopy -u=root -p=PASSWORDREMOVED db1 /backup/mysql/daily >> /var/log/mysql-hotcopy.log 2>&1
/usr/local/mysql/bin/mysqlhotcopy -u=root -p=PASSWORDREMOVED db2 /backup/mysql/daily >> /var/log/mysql-hotcopy.log 2>&1
/usr/local/mysql/bin/mysqlhotcopy -u=root -p=PASSWORDREMOVED db3 /backup/mysql/daily >> /var/log/mysql-hotcopy.log 2>&1
#
# Log the end time
echo 'END TIME: ' >> /var/log/mysql-hotcopy.log
date >> /var/log/mysql-hotcopy.log
echo '' >> /var/log/mysql-hotcopy.log;
#
# Now we tar the files
#
cd /backup/mysql/daily
/bin/tar -czvf mysqlbackup.tar.gz /backup/mysql/daily/*
#
# Chown the file
#
# FAILS HERE WHEN RUN FROM CRON
/bin/chown backupguy:backupguy /backup/mysql/daily/mysqlbackup.tar.gz >> /var/log/mysql-hotcopy.log 2>&1
#
# Now we rsync the tar file to the other server
#
echo '' >> /var/log/mysql-hotcopy.log
date >> /var/log/mysql-hotcopy.log
echo 'Rysnc Mysql Backup to HOSTNAMEREMOVED' >> /var/log/mysql-hotcopy.log
echo '--------------' >> /var/log/mysql-hotcopy.log
echo '' >> /var/log/mysql-hotcopy.log
rsync -avp -e "/usr/bin/ssh -l backupguy -i /home/users/backupguy/.ssh/DSA_KEY_REMOVED" /backup/mysql/daily/mysqlbackup.tar.gz backupguy@HOST.DOMAIN.COM:/backup/mysql/dc1web1 >> /var/log/mysql-hotcopy.log 2>&1
#
/bin/rm -rf /backup/mysql/archive/*
/bin/mv /backup/mysql/daily/mysqlbackup.tar.gz /backup/mysql/archive/
/bin/rm -rf /backup/mysql/daily/*
#

exit 0
0
Comment
Question by:Linux-Mechanic
  • 3
  • 3
  • 2
  • +1
9 Comments
 
LVL 7

Expert Comment

by:macker-
ID: 13840289
How very odd.  Have you tried inserting an echo statement between tar and chown, and I assume you never get the next line?  Does "rpm -V coreutils" come back clean?  Try running a memtest on it, just to make sure it's not the mysqldump operation filling memory and causing system instability, and/or an overheating CPU -- usually caused by a bad fan.
0
 
LVL 2

Expert Comment

by:Darshan_Jadav
ID: 13842242
give a bit of time between 2 instance of mysql backuo, insert a sleep 5, then fire another DB backup quire

0
 

Author Comment

by:Linux-Mechanic
ID: 13845465
Thanks for the suggestions guys.  The sleep idea is a good one.  rpm -V coreutils returns nothing.    I will try to run a memtest.  

The sleep was a good idea.  I put one in right before after the big mysql dump for 30 second and I put one in for 30 seconds after the tar operation.  Still no luck though.  I will try zipping and syncing seperately.  I'm also going to try not tarring them and rsyncing them as they are.

0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:Linux-Mechanic
ID: 13845865
SOB, the thing worked if I commented out the smallest DB, which is only 20K!  I really am at a lost.  There is no logical explanation for this.
0
 
LVL 7

Expert Comment

by:macker-
ID: 13846261
So, it sounds like it's working if you comment out _a_ database, as opposed to a specific database.  Is it possible there's a table lock issue somewhere?  Is it absolutely sure that the chown command is being run, but never exits?  Try doing "pstree -up" while the script is running, you should still see it executing as a child of CROND.
0
 

Author Comment

by:Linux-Mechanic
ID: 13847121
Doesn't appear to be a lock issue.  Log also lists that the tables have been unlocked again.   If I run the script manually then all DBs get backed up.  On the other hand it the DB I commented out is "mysql" that stores permissions and user names, I believe.  If so that may indicate some kind of lock issue, though I've backed up mysql on a bunch of other servers with no problems.  

I'm not sure if the chown command never exits.  It's not listed when I do a "pstree -up"  It records no errors in /var/log/messages or my log.  But I know the script fails there because the tar file gets created, yet it's still owned by root instead of "backupguy" and nothing happens after that.  It does not rsync the tar file or delete everything from the temp directory: /backup/mysql/daily.  No commands are executed after the tar command.  Even if I put an echo in the script after the tar command, it doesn't run.  It also doesn't seem like tar has errored out.  I've been able to untar the tar file that it's created before it exits.

Baffling.

I appreciate your responses.  I can tell by that fact that other people haven't responded that ithe problem is as strange as I thought.  I usually don't need to ask questions, so when I do they are often difficult.  

Frankly, I might just be happy with ignoring the "mysql" DB on this server, as long as I've got the big important DB backed up.  But my curiosity makes me want to know the answer and give someone credit for their help.  Since you've stuck with it Macker I'll probably just give the points to you, as at least I learned the pstree command, which I hadn't used before. :)  Thanks.

Thanks guys.
0
 
LVL 7

Accepted Solution

by:
macker- earned 2000 total points
ID: 13847868
In that case, it sounds not like it's failing on the chown command, but is failing to continue from the tar command.

I would say try redirecting both stderr and stdout for tar, possibly even strace it (though this would probably be very noisy, considering the operations)...  also, try adding -x to the bash command line at the start, that'll enable verbose output; which reminds me, does cron ever e-mail you regarding the job, does the bash session show as still running under crond - or just disappears?

The only difference I can think of is the environment variables that cron sets, which differs from a user's login environment.  You have obviously been careful to use fully defined paths everywhere.  This is quite perplexing.  I'm going to end up trying to recreate this over the weekend... :)
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 13852892
please insert before first and right after each mysqlhotcopy:

  /full/path/to/mysql -p YOURPASS status
  /full/path/to/mysqladmin -p YOURPASS status
  /full/path/to/mysqladmin -p YOURPASS extended-status

and redirect to a file, then check if there're changes (for example locks), think your are experianced enough to check the important ones ;-)
0
 
LVL 2

Expert Comment

by:Darshan_Jadav
ID: 13858551
Check if "core" files are there in yr Mysql DB dir, it messes up when dump is called up from srcipt, but a validation in script to del any "core" file before running backup
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses
Course of the Month12 days, 22 hours left to enroll

578 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question