How to execute (python) script in crontab with command line arguments

I have a python script that processes data files which I currently run from the Linux command line:
$ /path/process.py archive1.csv archive1 2> error.log
with arguments: name_of_file to process, subdirectory for the output file, and 2> error.log catches any errors
Each of 24 archives takes at least 60 seconds to process; I also have 24 data files which take 15 - 30 seconds each to process.  I've tried to get the python script to run in my crontab iteratively for each archive file but nothing happens.  I've tried integrating the command line arguments into a bash script with similar results, probably because I don't know enough about bash scripting.

The alternative to a bash script would be to change the python script so that it will run as is from the crontab, but that looks more difficult, so I thought I'd try this approach first.
 
sara_bellumAsked:
Who is Participating?
 
TintinConnect With a Mentor Commented:
I'd write a bash wrapper script, eg:
#!/bin/bash
for i in $(seq 1 24)
do
   /path/process.py archive$i.csv archive$i 2>>/path/to/error.log
done

Open in new window

0
 
omarfaridCommented:
to run crontab jobs,

- set env variables needed in the script
- use full path names to files, dirs, and commands
- change to required dir

you may have a script that will loop for each file to process



0
 
TintinCommented:
If your system doesn't have the seq command, then change the above to:
#!/bin/bash
i=1
 
while [ $i -le 24 ]
do
  /path/process.py archive$i.csv archive$i 2>>/path/to/error.log
  let i++
done

Open in new window

0
Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

 
sara_bellumAuthor Commented:
Thanks very much!  I have the seq command so the first bash wrapper script worked :)  I spent the better part of today trying to upload the results, but that would be a lot easier if all of my output files were in one directory.  So I wrote another bash wrapper (copied below) to move the 2 output files for each data1 - 24 folder from /path/data$i to /path/test/.  But this time (because I wrote it) the paths are not recognized - the error reads "cannot stat '/path/data$i/data$i_filename' No such file or directory" for each of 48 iterations.  Let me know what I'm doing wrong, or whether I should open another question, thanks.  

#!/bin/bash
 
SRC='/path1/data$i'
DST='/path2/test'
src1='data$i_1_minutes.csv'
src2='data$i_60_minutes.csv'
dst1='data$i_minutes.csv'
dst2='data$i_hours.csv'
for i in $(seq 1 24)
do
  mv $SRC/$src1 $DST/$dst1
  mv $SRC/$src2 $DST/$dst2
done

Open in new window

0
 
sara_bellumAuthor Commented:
I should note that the source and destination paths are correct, but the system isn't translating data$i into folder names data1 - data24, and so the filenames of course have the same problem.
0
 
TintinConnect With a Mentor Commented:
From your description and last script, you should just need:
#!/bin/bash
for i in $(seq 1 24)
do
  mv /path1/data$i/data${i}_minutes.csv /path2/test
  mv /path/data$i/data${i}_hours.csv /path2/test
done

Open in new window

0
 
sara_bellumAuthor Commented:
Brilliant!! I'm still confused as to why bash can't interpret $i when it's in a pre-defined variable (it would be nice to know why) but I'm very happy with this result, thanks again.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.