StephenMcGowan
asked on
Job specific file passing
Hi.
I have a shell script: pipeline.sh
I'd like to run the shell script 1000 times. This will be done by submitting each script to a linux server queue.
A TMP folder is set up for each job's files:
TMP=~/permanalysis/tmp.$$
mkdir $TMP
When multiple runs are taking place at the same time, there seems to be a problem with $queryfile being passed to fastacutter.pl.
I'm wondering if it is at all possible to give the created $queryfile a unique (job specific) name and then for this unique $queryfile to be used by fastacutter.pl?
I think it's currently erroring as multiple scripts have fastacutter.pl calling the same $queryfile.
pipeline.sh
fastacutter.txt
I have a shell script: pipeline.sh
I'd like to run the shell script 1000 times. This will be done by submitting each script to a linux server queue.
A TMP folder is set up for each job's files:
TMP=~/permanalysis/tmp.$$
mkdir $TMP
When multiple runs are taking place at the same time, there seems to be a problem with $queryfile being passed to fastacutter.pl.
I'm wondering if it is at all possible to give the created $queryfile a unique (job specific) name and then for this unique $queryfile to be used by fastacutter.pl?
I think it's currently erroring as multiple scripts have fastacutter.pl calling the same $queryfile.
pipeline.sh
fastacutter.txt
ASKER
Hi n4th4ne1ch,
Thanks for getting back to me.
"you have an old style of scripting; that is, you are scripting more for SH than for BASH."
This script isn't mine. I've been asked to modify it so that I can run multiple jobs/instances at the same time by submitting the jobs to a queue.
So far, I've tried running 10 instances of the script at the same time by submitting the jobs from the command line:
qsub -b y –cwd -N test1 sh pipeline.sh
job submitted
qsub -b y –cwd -N test2 sh pipeline.sh
job submitted
qsub -b y –cwd -N test3 sh pipeline.sh
job submitted
qsub -b y –cwd -N test4 sh pipeline.sh
job submitted
etc upto job10
For three of the jobs, the process worked fine and the job ran to completion. But for the other seven there seemed to be a problem passing the file to fastacutter.pl
I'm wondering if these jobs are conflicting and trying to using the same files.
I'd like everything for a job to be kept separately from another job ( i.e. "I'm wondering if it is at all possible to give the created $queryfile a unique (job specific) name and then for this unique $queryfile to be used by fastacutter.pl?") <- therefore keeping everything used by one job separate from another job.
"It appears you are running something concurrently which is all trying to open and read/use the same file and you want to somehow automatically generate filenames so each instance is only affecting its own file."
exactly. I have a script which has been written to run on it's own. But I've been asked to modify it so that it can be run 1000 times at the same time (without the jobs conflicting)
If this is achievable, what modifications would I need to make to this script?
Thanks,
Stephen.
Thanks for getting back to me.
"you have an old style of scripting; that is, you are scripting more for SH than for BASH."
This script isn't mine. I've been asked to modify it so that I can run multiple jobs/instances at the same time by submitting the jobs to a queue.
So far, I've tried running 10 instances of the script at the same time by submitting the jobs from the command line:
qsub -b y –cwd -N test1 sh pipeline.sh
job submitted
qsub -b y –cwd -N test2 sh pipeline.sh
job submitted
qsub -b y –cwd -N test3 sh pipeline.sh
job submitted
qsub -b y –cwd -N test4 sh pipeline.sh
job submitted
etc upto job10
For three of the jobs, the process worked fine and the job ran to completion. But for the other seven there seemed to be a problem passing the file to fastacutter.pl
I'm wondering if these jobs are conflicting and trying to using the same files.
I'd like everything for a job to be kept separately from another job ( i.e. "I'm wondering if it is at all possible to give the created $queryfile a unique (job specific) name and then for this unique $queryfile to be used by fastacutter.pl?") <- therefore keeping everything used by one job separate from another job.
"It appears you are running something concurrently which is all trying to open and read/use the same file and you want to somehow automatically generate filenames so each instance is only affecting its own file."
exactly. I have a script which has been written to run on it's own. But I've been asked to modify it so that it can be run 1000 times at the same time (without the jobs conflicting)
If this is achievable, what modifications would I need to make to this script?
Thanks,
Stephen.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I've requested that this question be deleted for the following reason:
Abandoned
Abandoned
This is bad etiquette. StephenMcGowan has not responded to my last post which I intended as a complete solution to his problem as asked
You can remove all of these things:
Open in new window
and simply use the default value option of bash:
Open in new window
Also you generally don't use ALL_CAPS_VARIABLES for variables that are only found within a single script. That is only a convention, there is no functional difference really.
As for your question, it's a little hard to understand what you're asking. Let me tell you what it appears you are asking, and hopefully my answer will help:
It appears you are running something concurrently which is all trying to open and read/use the same file and you want to somehow automatically generate filenames so each instance is only affecting its own file.
If that is correct you can do this:
filename=/my/normal/filena
In bash $$ will be the PID of the process, and therefore would be different for each instance.
Perl I believe has the same syntax for PID : $$