[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 664
  • Last Modified:

gzip cron job help

Hello,

I am a web developer and want to run a series of cron jobs but I don't know where to find the information on how to do each particular one...

1. I want to gzip a series of directories /root/userfiles/images /root/userfiles/images2 etc... but i want to encode some sense of the root structure so in the final gzip file it contains /root/userfiles/images/image1.jpg instead of image1.jpg do you know what I mean?

2. Secondly I want that gzipped file to be ftp'd or emailed or somehow transferred to me and deleted off the server...

We could probably break down question 1 into 2 steps of how do i gzip a series of directories using cron and how do i preserve directory structure while only including files in certain sub-directories

Thanks very much
:Ant
0
antum
Asked:
antum
4 Solutions
 
JJSmithCommented:


Do it all by hand - then script it - then put that script in a cron.

Notes:

gzip -N -r /root/userfiles/images
         # will compress all files in the directory saving the original filenames (use -N) on the gunzip to restore the original filename

Can't help on ftp or mail, obviously its possible but depends on setup and ftp and mail services at receving end.

Cheers
JJ
0
 
pjedmondCommented:
Probably easiest to create a bash script and tar the directories/files that you want:

--------8X----------
#!/bin/bash

tar czf ~/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2
mutt -a ~/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <"."
rm -Rf ~/tarredfile.tar.gz
--------8X----------

Reason for using tar is that it maintains file structure and does not delete the original (Not a good idea in some situations). Also, you can stick them all in one tar file:)

OK - there are loads of other ways of sending the email or transferring it using scp (slightly complicated to do this securely, but it can be done - Look here:

http://www.cvrti.utah.edu/~dustman/no-more-pw-ssh/

Now for the cron bit:

crontab -e, and add your extra crontab (We'll call the script sendit.sh). Obviously test that it works without cron before adding a cron entry:

0 23 * * 1-5 /root/scripts/sendit.pl #Transfer of data

This entry runs the script every Mon-Fri at 2300.

http://www.itworld.com/Comp/2378/swol-0825-unix101/

for more examples.

HTH:)

0
 
pjedmondCommented:
Ooops - should be sendit.sh in the crontab entry...but you spotted that 'deliberate mistake'...Didn't you?
0
NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

 
pjedmondCommented:
Obviously replace the tar line with the gzip line in the script if you prefer. I just happen to use tar for this type of thing. Youcould use bzip2 as well - feel free to play;)

man tar
man bzip2

etc

HTH:)
0
 
Duncan RoeSoftware DeveloperCommented:
You want to use tar to keep the file structure. You can tar all the directories into a single file or have a tar file for each dir, as you prefer. tar has a -z (gzip) option and newer ones have a -j (bzip2) option (bzip2 usually compresses even better than gzip). Have a look at the man page for tar and post again if you have more questions.
As for transferring your files: mail has the advantage over ftp that you don't have to enter a password or code one in a script, you just need to be sure that no mail server on the way will refuse to pass on the attachment owing to size, content or whatever. What mail clients do you have on the web servers? - you will use one of them on the command line. Experiment, post again if you run out of ideas. I'd recommend installing mutt if it's not there already.
0
 
ahoffmannCommented:
I'd improve pjedmod's script as follows:

--------8X----------
#!/bin/sh
/bin/tar czf /tmp/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2 && \
/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <"." && \
/bin/rm -Rf /tmp/tarredfile.tar.gz
--------8X----------
0
 
pjedmondCommented:
Yep - I'd agree with that:)

Always good to have someone check and improve things:) I'm going to have to work hard to improve the quality of my posts;)
0
 
Duncan RoeSoftware DeveloperCommented:
Is '<"."' an attempt to make a "here" document? Look what it does:
07:47:47$ cat <"."
cat: -: Is a directory
bash connected the current directory to standard input - what you asked it to do but not what you want

Take 3:-

--------8X----------
#!/bin/sh
/bin/tar czf /tmp/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2 && \
/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <<EOF
.
EOF
[ $? -eq 0 ] && /bin/rm -Rf /tmp/tarredfile.tar.gz
--------8X----------
0
 
pjedmondCommented:
I see your point, but if you do it that way, surely you can:

--------8X----------
#!/bin/sh
/bin/tar czf /tmp/tarredfile.tar.gz /root/userfiles/images /root/userfiles/images2 && \
/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com <"."
[ $? -eq 0 ] && /bin/rm -Rf /tmp/tarredfile.tar.gz
--------8X----------

0
 
pjedmondCommented:
Thought about this a bit more.

The meaning of "." depends on the context.

As you quite rightly said:

cat <"."

does try and cat the current directory, because cat expects a filename, and hence "." is assumed to be a filename.

. filename

executes a filename.

Under these circumstances (the use of mutt), I'd always considered the <"." to be perfectly valid(as it works), as I'd assumed that the . concerned was the 'finalising' of the email, which it effectively does....but if you were to change this to <"hello there\n". From a 'purists' perspective, I like your apporach more. Always good to have someone 'red penning' my work ;)

(   (()
(`-' _\
 ''  ''
0
 
Duncan RoeSoftware DeveloperCommented:
Hi pjedmond - I think you must have been lucky with mutt - it would have been given a directory as its input and chose not to complain.
cat does not "expect a filename" - the man page documents that in the absence of an argument, it will read from standard input. The construct '<"."' tells the shell (bash, say) to connect "." (the current directory) to the standard input of the program you told it to run (cat or mutt). It seems that mutt chooses to treat the error as EOF (it could just be treating a -1 return from fgets as EOF, I haven't checked).
20:20:27$ cat <"hello there\n"
-bash: hello there\n: No such file or directory
Notice that error comes from *bash* - cat or mutt never gets started. "." is a valid existing filesystem entity, albeit a directory, so bash doesn't complain.
0
 
pjedmondCommented:
cat does not "expect a filename" - Ok - I'll expand - If given a name it assumes it to be a filename.

I was wondering if this works due to the first 2 'files' in the current directory are:

.
..

Based on the fact that a '.' on its own terminates the email message when telnetting to an SMTP server, hence the email is completed by the first '.' on it's own. However, as you've suggested, it appears that the error produced *really* is interpreted as the EOT! EEeeeek scary - guaranteed to break unexpectedly during an upgrade! I've tweaked the code on a few of my customers to correct (even though I've used this hack for years! I also accept that it is a horrible hack, and I also admit having used it for years on the basis that it it just 'worked'.

How about going for the more conventional then:

/bin/mutt -a /tmp/tarredfile.tar.gz -s "Tarred files" me@mydomain.com < /home/me/messagefile

where messagefile is:

-------8X-------------------
Here are your tarred files
-------8X-------------------

Wow - best question I've participated in this month!

(   (()
(`-' _\
 ''  ''


0
 
pjedmondCommented:
mutt -a ~/list.txt -s "Tarred files" me@cb.ws < /dev/null

also works:) ......

(   (()
(`-' _\
 ''  ''

0
 
Duncan RoeSoftware DeveloperCommented:
</dev/null is fine. Gives an immediate EOF (and *no* error ;)
0
 
antumAuthor Commented:
Thankyou all very much for your help!
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Tackle projects and never again get stuck behind a technical roadblock.
Join Now