How to execute (python) script in crontab with command line arguments

I have a python script that processes data files which I currently run from the Linux command line:
$ /path/process.py archive1.csv archive1 2> error.log
with arguments: name_of_file to process, subdirectory for the output file, and 2> error.log catches any errors
Each of 24 archives takes at least 60 seconds to process; I also have 24 data files which take 15 - 30 seconds each to process.  I've tried to get the python script to run in my crontab iteratively for each archive file but nothing happens.  I've tried integrating the command line arguments into a bash script with similar results, probably because I don't know enough about bash scripting.

The alternative to a bash script would be to change the python script so that it will run as is from the crontab, but that looks more difficult, so I thought I'd try this approach first.
 
sara_bellumAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

omarfaridCommented:
to run crontab jobs,

- set env variables needed in the script
- use full path names to files, dirs, and commands
- change to required dir

you may have a script that will loop for each file to process



0
TintinCommented:
I'd write a bash wrapper script, eg:
#!/bin/bash
for i in $(seq 1 24)
do
   /path/process.py archive$i.csv archive$i 2>>/path/to/error.log
done

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
TintinCommented:
If your system doesn't have the seq command, then change the above to:
#!/bin/bash
i=1
 
while [ $i -le 24 ]
do
  /path/process.py archive$i.csv archive$i 2>>/path/to/error.log
  let i++
done

Open in new window

0
Fundamentals of JavaScript

Learn the fundamentals of the popular programming language JavaScript so that you can explore the realm of web development.

sara_bellumAuthor Commented:
Thanks very much!  I have the seq command so the first bash wrapper script worked :)  I spent the better part of today trying to upload the results, but that would be a lot easier if all of my output files were in one directory.  So I wrote another bash wrapper (copied below) to move the 2 output files for each data1 - 24 folder from /path/data$i to /path/test/.  But this time (because I wrote it) the paths are not recognized - the error reads "cannot stat '/path/data$i/data$i_filename' No such file or directory" for each of 48 iterations.  Let me know what I'm doing wrong, or whether I should open another question, thanks.  

#!/bin/bash
 
SRC='/path1/data$i'
DST='/path2/test'
src1='data$i_1_minutes.csv'
src2='data$i_60_minutes.csv'
dst1='data$i_minutes.csv'
dst2='data$i_hours.csv'
for i in $(seq 1 24)
do
  mv $SRC/$src1 $DST/$dst1
  mv $SRC/$src2 $DST/$dst2
done

Open in new window

0
sara_bellumAuthor Commented:
I should note that the source and destination paths are correct, but the system isn't translating data$i into folder names data1 - data24, and so the filenames of course have the same problem.
0
TintinCommented:
From your description and last script, you should just need:
#!/bin/bash
for i in $(seq 1 24)
do
  mv /path1/data$i/data${i}_minutes.csv /path2/test
  mv /path/data$i/data${i}_hours.csv /path2/test
done

Open in new window

0
sara_bellumAuthor Commented:
Brilliant!! I'm still confused as to why bash can't interpret $i when it's in a pre-defined variable (it would be nice to know why) but I'm very happy with this result, thanks again.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.