looping in a bash script

I have been doing this for hours and I have just given up:...

The task: loop through a folder of files and perform 6 scripts on each file to run as one shell script

Each time a file is processed by one script it gets put in another folder. The intention is to pass the resulting file name to the next script. I dont want to keep the folder path and the original .txt or whatever just the filename and the new file extension. Can anyone help?

The scripts are as follows (on a mac as a shell script)

for each file as $file

$ gzcat $file.fq.gz | ./reformat_sequence_data.rb > ./reformatsequence/$file.txt

$ python qualityMask.py ./reformatsequence/$file.txt  ./qualitymask/$file.seq 20 1

$ ./unique_seq_counts.rb ./qualitymask/$file.seq > ./uniquecounts/$file.counts.txt

$ cut -f1 ./uniquecounts/$file.counts.txt > ./uniqueseq/$file.uniq.seq

$ ./bowtie-1.1.0/bowtie -r -m1 -v1  ./bowtie-1.1.0/indexes/hg19 ./uniqueseq/$file.uniq.seq > ./alignedunique/$file.bowtie.txt
LVL 1
sebastizzAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

omarfaridCommented:
try this

cd /my/path
for file in `ls filename*` # put your criteria in place of filename* e.g. myfile*.ext
do
    gzcat $file.fq.gz | ./reformat_sequence_data.rb > ./reformatsequence/$file.txt
    python qualityMask.py ./reformatsequence/$file.txt  ./qualitymask/$file.seq 20 1
   ./unique_seq_counts.rb ./qualitymask/$file.seq > ./uniquecounts/$file.counts.txt
   cut -f1 ./uniquecounts/$file.counts.txt > ./uniqueseq/$file.uniq.seq
   ./bowtie-1.1.0/bowtie -r -m1 -v1  ./bowtie-1.1.0/indexes/hg19 ./uniqueseq/$file.uniq.seq > ./alignedunique/$file.bowtie.txt
done

I assumed that you change to the dir where the files are and where your scripts and other folders are
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
TintinCommented:
no need for the ls

for file in filename.*

Open in new window

0
sebastizzAuthor Commented:
so does

for file in filename.*

mean for every file in the directory I specify as that would be ideal
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

sebastizzAuthor Commented:
so ive got as far as the code below. However it stalls at the python bit. The error says:

Traceback (most recent call last):
  File "qualityMask.py", line 6, in <module>
    iFile = open(r'%s' % sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: './reformatsequence/SLX-8866.25%_A.000000000-A9AMY.s_1.r_1.fq.gz.txt.txt'
./unique_seq_counts.rb:3:in `each_line': Is a directory - ./qualitymask/ (Errno::EISDIR)
      from ./unique_seq_counts.rb:3:in `each'
      from ./unique_seq_counts.rb:3:in `<main>'

#!/bin/bash
shopt -s extglob

for f in ./Zipped/*; do

gzcat $f | ./reformat_sequence_data.rb > $f.txt

file=$(basename $f)
python qualityMask.py ./reformatsequence/$file.txt  ./qualitymask/$file.seq 20 1

./unique_seq_counts.rb ./qualitymask/$fileseq > ./uniquecounts/$file.counts.txt

cut -f1 ./uniquecounts/$file.counts.txt > ./uniqueseq/$file.uniq.seq

./bowtie-1.1.0/bowtie -r -m1 -v1  ./bowtie-1.1.0/indexes/hg19 ./uniqueseq/$file.uniq.seq > ./alignedunique/$file.bowtie.txt

done

Open in new window

0
tel2Commented:
Hi sebastizz,

Regarding your post #40237238:

> so does
> for file in filename.*
> mean for every file in the directory I specify as that would be ideal


Not quite.  It means every file starting with 'filename.'.  Change the word 'filename' to anything you like, or change the template to match the files you want to match (e.g. '*.gz' to match all files with a 'gz' extension).
0
TintinCommented:
if you want all files do

for file in *

If you have subdirectories, then change to

for file in $(find . -maxdepth 1 -type f)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.