[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Crontab Output

Posted on 2007-07-25
17
Medium Priority
?
548 Views
Last Modified: 2013-12-16
I'm running a FreeBSD 6.0 web server.

Currently, I have several crob jobs that run nightly. They go out and ncftpget files and download them to the server and process them. Part of a nightly site update routine.

I get the output of each cron job emailed to me. Everything is working as it should, however, I am not satisfied with the output of the cron jobs due to the fact the emails are quite large.

An few snippet examples of an email is :

_07252007_013000.zip:              ETA:   0:37  123.79/132.64 MB  247.74 kB/s
_07252007_013000.zip:              ETA:   0:36  123.88/132.64 MB  247.42 kB/s
_07252007_013000.zip:              ETA:   0:36  124.02/132.64 MB  247.24 kB/s
_07252007_013000.zip:              ETA:   0:35  124.16/132.64 MB  247.00 kB/s
_07252007_013000.zip:              ETA:   0:34  124.36/132.64 MB  246.94 kB/s
_07252007_013000.zip:              ETA:   0:33  124.57/132.64 MB  246.88 kB/s
_07252007_013000.zip:              ETA:   0:32  124.81/132.64 MB  246.86 kB/s
_07252007_013000.zip:              ETA:   0:32  125.04/132.64 MB  246.86 kB/s
_07252007_013000.zip:              ETA:   0:30  125.34/132.64 MB  246.98 kB/s
_07252007_013000.zip:              ETA:   0:29  125.53/132.64 MB  246.89 kB/s
_07252007_013000.zip:              ETA:   0:29  125.63/132.64 MB  246.60 kB/s
_07252007_013000.zip:              ETA:   0:29  125.76/1!
 32.64 MB  246.37 kB/s
_07252007_013000.zip:              ETA:   0:28  125.94/132.64 MB  246.26 kB/s
_07252007_013000.zip:              ETA:   0:27  126.18/132.64 MB  246.26 kB/s
_07252007_013000.zip:              ETA:   0:26  126.47/132.64 MB  246.35 kB/s
_07252007_013000.zip:              ETA:   0:24  126.81/132.64 MB  246.56 kB/s
_07252007_013000.zip:              ETA:   0:23  127.17/132.64 MB  246.79 kB/s
_07252007_013000.zip:              ETA:   0:21  127.61/132.64 MB  247.17 kB/s
_07252007_013000.zip:              ETA:   0:19  128.05/132.64 MB  247.56 kB/s
_07252007_013000.zip:              ETA:   0:17  128.45/132.64 MB  247.54 kB/s
_07252007_013000.zip:              ETA:   0:17  128.45/132.64 MB  247.38 kB/s
_07252007_013000.zip:              ETA:   0:17  128.53/132.64 MB  247.08 kB/s
_07252007_013000.zip:              ETA:   0:17  128.65/132.64 MB  246.86 kB/s
_07252007_013000.zip:              ETA:   0:16  128.83/132.64 MB  246.73 kB/s
_072!
 52007_013000.zip:              ETA:   0:15  129.06/132.64 MB  !
 246.72 k
B/s
_07252007_013000.zip:              ETA:   0:14  129.35/132.64 MB  246.81 kB/s
_07252007_013000.zip:              ETA:   0:12  129.69/132.64 MB  246.98 kB/s
_07252007_013000.zip:              ETA:   0:11  130.06/132.64 MB  247.25 kB/s
_07252007_013000.zip:              ETA:   0:09  130.50/132.64 MB  247.62 kB/s
_07252007_013000.zip:              ETA:   0:07  130.95/132.64 MB  248.01 kB/s
_07252007_013000.zip:              ETA:   0:05  131.35/132.64 MB  248.30 kB/s
_07252007_013000.zip:              ETA:   0:04  131.65/132.64 MB  248.42 kB/s
_07252007_013000.zip:              ETA:   0:03  131.99/132.64 MB  248.46 kB/s
_07252007_013000.zip:              ETA:   0:03  132.00/132.64 MB  248.06 kB/s
_07252007_013000.zip:              ETA:   0:02  132.05/132.64 MB  247.76 kB/s
_07252007_013000.zip:              ETA:   0:02  132.09/132.64 MB  247.40 kB/s
_07252007_013000.zip:              ETA:   0:02  132.17/132.64 MB  247.11 kB/s
_07252007_013000.zip:  !
            ETA:   0:01  132.28/132.64 MB  246.86 kB/s
_07252007_013000.zip:              ETA:   0:01  132.40/132.64 MB  246.64 kB/s
_07252007_013000.zip:              ETA:   0:00  132.55/132.64 MB  246.47 kB/s
_07252007_013000.zip:              ETA:   0:00  132.64/132.64 MB  246.32 kB/s
_07252007_013000.zip:                                  132.64 MB  246.32 kB/s
_07252007_013000.zip:                                  132.64 MB  246.32 kB/s
Archive:  _07252007_013000.zip
replace INCP_813C.csv? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL
(assuming [N]one)
 extracting: LOTL_8151.zip
 extracting: RESI_8156.zip
Archive:  RESI_8156.zip
 extracting: 10028502.jpg
 extracting: 10033724.jpg
 extracting: 10033728.jpg
 extracting: 10033729.jpg
 extracting: 10033765.jpg
 extracting: 10038788.jpg
 extracting: 10038788_01.jpg
 extracting: 10038788_02.jpg
 extracting: 10038788_03.jpg
 extracting: 10038788_04.jpg
 extracting: 10038788_05.jpg
 extracting: 10038788_06.jpg
 extracting: 10038788_07.jpg
 extracting: 10039396.jpg
 extracting: 10039396_01.jpg
 extracting: 10039396_02.jpg
 extracting: 10039396_03.jpg
 extracting: 10039396_04.jpg
 extracting: 10039396_05.jpg
 extracting: 10039396_06.jpg
 extracting: 10039396_07.jpg
 extracting: 10039396_08.jpg
 extracting: 10039396_09.jpg
 extracting: 10039396_10.jpg
 extracting: 10039396_11.jpg
 extracting: 10039396_12.jpg
 extracting: 10039396_13.jpg
 extracting: 10039396_14.jpg
 extracting: 10039474.jpg
 extracting: 10041147.jpg
 extracting: 10041147_01.jpg
 extracting: 10041147_02.jpg
 extracting: 10041147_03.jpg
 extracting: 10041147_04.jpg
 extracting: 10041147_05.jpg
 extracting: 10041147_06.jpg
 extracting: 10041147_07.jpg
 extracting: 10041147_08.jpg
 extracting: 10041147_09.jpg
 extracting: 10044941.jpg
 extracting: 10045082.jpg
 extracting: 10045419.jpg
.
.
.
 extracting: 10055319_08.jpg
 extracting: 10055319_09.jpg
 extracting: RESI_8153.csv



RESI_LINC.csv. updated to database
rm: /var/www/IDX/lincoln/COMI_*.zip: No such file or directory
rm: /var/www/IDX/lincoln/INCP_*.zip: No such file or directory
/var/www/IDX/lincoln/ftp_lincoln_rapattoni[90]: path: not found

**************************************
END SNIPPETS



What I would like to have happen .. is that A) I don't want to see the lines with the ETA while the file is downloading output to the email and B) I don't need to see the extract list of files. Basically, I only want any text output that is programmed into the script (e.g. RESI_LINC.csv. updated to database) and any error messages that are generated.

If you have any questions or need clarification to what I'm looking for, plz let me know.

0
Comment
Question by:Zack Soderquist
  • 11
  • 3
  • 2
  • +1
17 Comments
 
LVL 85

Expert Comment

by:ozo
ID: 19568405
where in the script are you generating the ETA and  extract lists?
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569003
I'm not ...I think crontab is doing it
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569009
I'm not ...I think crontab is doing it
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569029
Here is the script


#!/usr/local/bin/ksh

########################
# Connection Variables #
########################

user="xxx"
pass="xxx"
ftphost="xxx.xxxx.xxx"

####################
# System Variables #
####################

ncftpget="/usr/local/bin/ncftpget"
lynx="/usr/local/bin/lynx"
lowercase="/usr/local/bin/lowercase"
nophoto="/var/www/IDX/no_photo.jpg"
unzip="/usr/local/bin/unzip"
picpath="/var/www/IDX/beatrice/pics/"
path="/var/www/IDX/beatrice/"




########################
# Beatrice Residential #
########################

        ##################
        # File Variables #
        ##################

                ###################################################################
                ### Create Symbolic Link in Root of Site Named IDX              ###
                ### Target is /var/www/IDX/city/ [ln -s /var/www/IDX/city/ IDX] ###
                ### If multiple cities target create IDX directory and symbolic ###
                ### links to each city.                                         ###
                ###################################################################

        dbupdate="http://www.xxxx.com/xxx/xxxx/RSupdate.php"
        zipfile="listings-residential.txt.gz"
        datafile="listings-residential.txt"
        date=`date +%Y%m%d`
        date2=$(($date-1))
        tarfile="pics-residential-$date2.tar"

        ################
        # Begin Script #
        ################

        ${ncftpget} -E -u "${user}" -p ${pass} ${ftphost} . idx/${zipfile}      # Download File(s)

        if [ $? = 0 ]
          then
              gunzip -d ${zipfile}                                      # Unzip Downloaded File
              mv -f ${datafile} ${path}                                 # Move Data File to Main Folder
              chmod 777 ${path}${datafile}                              # Change Permissions of Data File

            wait

            ${lynx} -dump ${dbupdate}                                   # Use Lynx to Update the DB
              if [ $? = 0 ]
                then
                  echo ${datafile}." updated to database"
                  rm ${path}${datafile}                                 # Remove the Data File
                else
                  echo ${datafile}." not updated to database"
              fi
          else
            echo "ERROR: ".${zipfile}." not available for download"
        fi


        cd ${picpath}                                                   # Move to the Pics Folder
        ${ncftpget} -E -u "${user}" -p ${pass} ${ftphost} . idx/${tarfile}      # Download File(s)

        if [ $? = 0 ]
          then
            chmod 777 ${picpath}${tarfile}                              # Change Permissions of TAR File
            tar -xf ${tarfile}                                          # Untar TAR File
            wait
            rm ${tarfile}                                               # Remove TAR File
            cp ${nophoto} ${path}pics/                                  # Copy the No Photo Image
            find . -type f|xargs chmod 755                              # Change Permissions of Photos
            ${lowercase}                                                # Change All Photos to Lowercase
            echo ${tarfile}." downloaded and updated"
          else
            echo "ERROR: ".${tarfile}." not available for download"
        fi

convert="http://www.xxxxx.com/xxx/xxx/convert.php"
${lynx} -dump ${convert}
0
 
LVL 85

Expert Comment

by:ozo
ID: 19569075
gunzip -d ${zipfile}   2> /dev/null

0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569082
huh?
0
 
LVL 85

Accepted Solution

by:
ozo earned 2000 total points
ID: 19569086
${ncftpget} -E -u "${user}" -p ${pass} ${ftphost} . idx/${zipfile} 2> /dev/null
0
 
LVL 62

Expert Comment

by:gheist
ID: 19569102
Look inside /etc/crontab and /var/cron/tabs/

Output you describe looks like
1) "fetch" fetching come file
2) "unzip" extracting the file

I need no sniplet. I need headers of email.
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569131
My crontab

# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.yl0Z07yeBw installed on Wed Jul 25 10:59:28 2007)
# (Cron version -- $FreeBSD: src/usr.sbin/cron/crontab/crontab.c,v 1.19 2002/10/01 22:59:11 dd Exp $)
00 06 * * * /var/www/IDX/omaha/ftp_omaha_rapattoni
50 07 * * * /var/www/IDXscripts/ftp_norfolk_fnis
#50 07 * * 1 /var/www/IDXscripts/ftp_norfolk_fnis_monday
00 11 * * * /var/www/IDX/lincoln/ftp_lincoln_rapattoni
#50 22 * * 2-5 /var/www/IDX/councilbluffs/ftp_councilbluffs_fnis
#50 22 * * 1 /var/www/IDX/councilbluffs/ftp_councilbluffs_fnis_monday
#50 20 * * 2-5 /var/www/IDX/grandisland/ftp_grandisland_fnis
#50 20 * * 1 /var/www/IDX/grandisland/ftp_grandisland_fnis_monday
#50 21 * * * /var/www/IDX/beatrice/ftp_beatrice_fnis
~
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569195
Header of email

Delivered-To: xxxxxxx@gmail.com
Received: by 10.114.66.7 with SMTP id o7cs810080waa;
        Wed, 25 Jul 2007 08:51:53 -0700 (PDT)
Received: by 10.90.52.18 with SMTP id z18mr669570agz.1185378712023;
        Wed, 25 Jul 2007 08:51:52 -0700 (PDT)
Return-Path: <xxxx@xxxx.xxxxxx.com>
Received: from xxxx.xxxxxx.com (rrcs-88-88-88-88.west.xx.xxxx.com [88.88.88.88])
        by mx.google.com with ESMTP id 15si2923684nzp.2007.07.25.08.51.46;
        Wed, 25 Jul 2007 08:51:51 -0700 (PDT)
Received-SPF: neutral (google.com: 88.88.88.88 is neither permitted nor denied by best guess record for domain of xxxx@xxxx.xxxxxx.com)
Received: from xxxx.xxxxxx.com (localhost.xxxxxx.com [127.0.0.1])
      by xxxx.xxxxxx.com (8.13.3/8.13.3) with ESMTP id l6PG9djZ013358
      for <xxx@xxxx.xxxxxx.com>; Wed, 25 Jul 2007 11:09:39 -0500 (CDT)
      (envelope-from xxxx@xxxx.xxxxxx.com)
Received: (from xxxx@localhost)
      by xxxx.xxxxxx.com (8.13.3/8.13.3/Submit) id l6PG07QC013328;
      Wed, 25 Jul 2007 11:00:07 -0500 (CDT)
      (envelope-from xxxx)
Date: Wed, 25 Jul 2007 11:00:07 -0500 (CDT)
Message-Id: <200707251600.l6PG07QC013328@xxxx.xxxxxx.com>
From: xxxx@xxxx.xxxxxx.com (Cron Daemon)
To: xxxx@xxxxx.xxxxxx.com
Subject: Cron <xxxx@xxxxx> /var/www/IDX/lincoln/ftp_lincoln_rapattoni
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/www>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=xxxx>
X-Cron-Env: <USER=xxxx>
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569203
ozo .. what is 2> /dev/null mean
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569249
ok .. looked up 2> /dev/null

2> /dev/null supresses stderr
and
> /dev/null supresses stdout

would I not then want to use > /dev/null to surpress the output and not the errors. Cause I do want to know the errors if there are any.. and I don't want the output of it's good
0
 
LVL 62

Expert Comment

by:gheist
ID: 19569382
You concealed user who sent mail. You are on your own.
Examine script mentioned in beginning of mail message sent.

2>/dev/null means redirecting error output to nowhere.
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569419
the user is my user account on the server .. which is why it is concealed
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19569506
My server sends the results of my cron jobs to my email address..

In that email .. I get (see example above) a bunch of lines to indicate the download progress of an NCFTPGET command. I want to supress this download progress information, unless there is an error and I do want to know the error.

The second part is part of a gunzip extract command in which it echos out each file it is unzipping from the archive.. I do not want a list of all the files that it decompresses
0
 
LVL 10

Expert Comment

by:ssvl
ID: 19571879
put this at the end of the cron entry for the script  "   > /dev/null  " .

it is emails only errors now.  
0
 
LVL 7

Author Comment

by:Zack Soderquist
ID: 19636311
Ok .. got it fixed/figured out

Ozo's suggestion for the download worked well
${ncftpget} -E -u "${user}" -p ${pass} ${ftphost} . idx/${zipfile} 2> /dev/null

Trying to use just >/dev/null did not supress the download time but the 2>/dev/null worked
thanks Ozo

I found that using 'unzip -qL' solved my issue with the extract file list

For the extraction
I used
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The purpose of this article is to demonstrate how we can use conditional statements using Python.
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month18 days, 13 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question