Solved

gzip cron job help

Posted on 2006-06-20
15
584 Views
Last Modified: 2006-11-18
Hello,

I am a web developer and want to run a series of cron jobs but I don't know where to find the information on how to do each particular one...

1. I want to gzip a series of directories /root/userfiles/images /root/userfiles/images2 etc... but i want to encode some sense of the root structure so in the final gzip file it contains /root/userfiles/images/image1.jpg instead of image1.jpg do you know what I mean?

2. Secondly I want that gzipped file to be ftp'd or emailed or somehow transferred to me and deleted off the server...

We could probably break down question 1 into 2 steps of how do i gzip a series of directories using cron and how do i preserve directory structure while only including files in certain sub-directories

Thanks very much
:Ant
0
Comment
Question by:antum
15 Comments
 
LVL 6

Assisted Solution

by:JJSmith
JJSmith earned 50 total points
Comment Utility


Do it all by hand - then script it - then put that script in a cron.

Notes:

gzip -N -r /root/userfiles/images
         # will compress all files in the directory saving the original filenames (use -N) on the gunzip to restore the original filename

Can't help on ftp or mail, obviously its possible but depends on setup and ftp and mail services at receving end.

Cheers
JJ
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 200 total points
Comment Utility
Probably easiest to create a bash script and tar the directories/files that you want:

--------8X----------
#!/bin/bash

tar czf ~/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2
mutt -a ~/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <"."
rm -Rf ~/tarredfile.tar.gz
--------8X----------

Reason for using tar is that it maintains file structure and does not delete the original (Not a good idea in some situations). Also, you can stick them all in one tar file:)

OK - there are loads of other ways of sending the email or transferring it using scp (slightly complicated to do this securely, but it can be done - Look here:

http://www.cvrti.utah.edu/~dustman/no-more-pw-ssh/

Now for the cron bit:

crontab -e, and add your extra crontab (We'll call the script sendit.sh). Obviously test that it works without cron before adding a cron entry:

0 23 * * 1-5 /root/scripts/sendit.pl #Transfer of data

This entry runs the script every Mon-Fri at 2300.

http://www.itworld.com/Comp/2378/swol-0825-unix101/

for more examples.

HTH:)

0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
Ooops - should be sendit.sh in the crontab entry...but you spotted that 'deliberate mistake'...Didn't you?
0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
Obviously replace the tar line with the gzip line in the script if you prefer. I just happen to use tar for this type of thing. Youcould use bzip2 as well - feel free to play;)

man tar
man bzip2

etc

HTH:)
0
 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 150 total points
Comment Utility
You want to use tar to keep the file structure. You can tar all the directories into a single file or have a tar file for each dir, as you prefer. tar has a -z (gzip) option and newer ones have a -j (bzip2) option (bzip2 usually compresses even better than gzip). Have a look at the man page for tar and post again if you have more questions.
As for transferring your files: mail has the advantage over ftp that you don't have to enter a password or code one in a script, you just need to be sure that no mail server on the way will refuse to pass on the attachment owing to size, content or whatever. What mail clients do you have on the web servers? - you will use one of them on the command line. Experiment, post again if you run out of ideas. I'd recommend installing mutt if it's not there already.
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 100 total points
Comment Utility
I'd improve pjedmod's script as follows:

--------8X----------
#!/bin/sh
/bin/tar czf /tmp/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2 && \
/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <"." && \
/bin/rm -Rf /tmp/tarredfile.tar.gz
--------8X----------
0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
Yep - I'd agree with that:)

Always good to have someone check and improve things:) I'm going to have to work hard to improve the quality of my posts;)
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
Is '<"."' an attempt to make a "here" document? Look what it does:
07:47:47$ cat <"."
cat: -: Is a directory
bash connected the current directory to standard input - what you asked it to do but not what you want

Take 3:-

--------8X----------
#!/bin/sh
/bin/tar czf /tmp/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2 && \
/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <<EOF
.
EOF
[ $? -eq 0 ] && /bin/rm -Rf /tmp/tarredfile.tar.gz
--------8X----------
0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
I see your point, but if you do it that way, surely you can:

--------8X----------
#!/bin/sh
/bin/tar czf /tmp/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2 && \
/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <"."
[ $? -eq 0 ] && /bin/rm -Rf /tmp/tarredfile.tar.gz
--------8X----------

0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
Thought about this a bit more.

The meaning of "." depends on the context.

As you quite rightly said:

cat <"."

does try and cat the current directory, because cat expects a filename, and hence "." is assumed to be a filename.

. filename

executes a filename.

Under these circumstances (the use of mutt), I'd always considered the <"." to be perfectly valid(as it works), as I'd assumed that the . concerned was the 'finalising' of the email, which it effectively does....but if you were to change this to <"hello there\n". From a 'purists' perspective, I like your apporach more. Always good to have someone 'red penning' my work ;)

(   (()
(`-' _\
 ''  ''
0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
Hi pjedmond - I think you must have been lucky with mutt - it would have been given a directory as its input and chose not to complain.
cat does not "expect a filename" - the man page documents that in the absence of an argument, it will read from standard input. The construct '<"."' tells the shell (bash, say) to connect "." (the current directory) to the standard input of the program you told it to run (cat or mutt). It seems that mutt chooses to treat the error as EOF (it could just be treating a -1 return from fgets as EOF, I haven't checked).
20:20:27$ cat <"hello there\n"
-bash: hello there\n: No such file or directory
Notice that error comes from *bash* - cat or mutt never gets started. "." is a valid existing filesystem entity, albeit a directory, so bash doesn't complain.
0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
cat does not "expect a filename" - Ok - I'll expand - If given a name it assumes it to be a filename.

I was wondering if this works due to the first 2 'files' in the current directory are:

.
..

Based on the fact that a '.' on its own terminates the email message when telnetting to an SMTP server, hence the email is completed by the first '.' on it's own. However, as you've suggested, it appears that the error produced *really* is interpreted as the EOT! EEeeeek scary - guaranteed to break unexpectedly during an upgrade! I've tweaked the code on a few of my customers to correct (even though I've used this hack for years! I also accept that it is a horrible hack, and I also admit having used it for years on the basis that it it just 'worked'.

How about going for the more conventional then:

/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com < /home/me/messagefile

where messagefile is:

-------8X-------------------
Here are your tarred files
-------8X-------------------

Wow - best question I've participated in this month!

(   (()
(`-' _\
 ''  ''


0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
mutt -a ~/list.txt -s "Tarred files" me@cb.ws < /dev/null

also works:) ......

(   (()
(`-' _\
 ''  ''

0
 
LVL 34

Expert Comment

by:Duncan Roe
Comment Utility
</dev/null is fine. Gives an immediate EOF (and *no* error ;)
0
 

Author Comment

by:antum
Comment Utility
Thankyou all very much for your help!
0

Featured Post

Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

Join & Write a Comment

This is the error message I got (CODE) Error caused by incompatible libmp3lame 3.98-2 with ffmpeg I've googled this error message and found out sometimes it attaches this note "can be treated with downgrade libmp3lame to version 3.97 or 3.98" …
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now