looping in a bash script

Posted on 2014-08-02
Last Modified: 2014-08-17
I have been doing this for hours and I have just given up:...

The task: loop through a folder of files and perform 6 scripts on each file to run as one shell script

Each time a file is processed by one script it gets put in another folder. The intention is to pass the resulting file name to the next script. I dont want to keep the folder path and the original .txt or whatever just the filename and the new file extension. Can anyone help?

The scripts are as follows (on a mac as a shell script)

for each file as $file

$ gzcat $file.fq.gz | ./reformat_sequence_data.rb > ./reformatsequence/$file.txt

$ python ./reformatsequence/$file.txt  ./qualitymask/$file.seq 20 1

$ ./unique_seq_counts.rb ./qualitymask/$file.seq > ./uniquecounts/$file.counts.txt

$ cut -f1 ./uniquecounts/$file.counts.txt > ./uniqueseq/$file.uniq.seq

$ ./bowtie-1.1.0/bowtie -r -m1 -v1  ./bowtie-1.1.0/indexes/hg19 ./uniqueseq/$file.uniq.seq > ./alignedunique/$file.bowtie.txt
Question by:sebastizz
    LVL 40

    Accepted Solution

    try this

    cd /my/path
    for file in `ls filename*` # put your criteria in place of filename* e.g. myfile*.ext
        gzcat $file.fq.gz | ./reformat_sequence_data.rb > ./reformatsequence/$file.txt
        python ./reformatsequence/$file.txt  ./qualitymask/$file.seq 20 1
       ./unique_seq_counts.rb ./qualitymask/$file.seq > ./uniquecounts/$file.counts.txt
       cut -f1 ./uniquecounts/$file.counts.txt > ./uniqueseq/$file.uniq.seq
       ./bowtie-1.1.0/bowtie -r -m1 -v1  ./bowtie-1.1.0/indexes/hg19 ./uniqueseq/$file.uniq.seq > ./alignedunique/$file.bowtie.txt

    I assumed that you change to the dir where the files are and where your scripts and other folders are
    LVL 48

    Expert Comment

    no need for the ls

    for file in filename.*

    Open in new window

    LVL 1

    Author Comment

    so does

    for file in filename.*

    mean for every file in the directory I specify as that would be ideal
    LVL 1

    Author Comment

    so ive got as far as the code below. However it stalls at the python bit. The error says:

    Traceback (most recent call last):
      File "", line 6, in <module>
        iFile = open(r'%s' % sys.argv[1], 'r')
    IOError: [Errno 2] No such file or directory: './reformatsequence/SLX-8866.25%_A.000000000-A9AMY.s_1.r_1.fq.gz.txt.txt'
    ./unique_seq_counts.rb:3:in `each_line': Is a directory - ./qualitymask/ (Errno::EISDIR)
          from ./unique_seq_counts.rb:3:in `each'
          from ./unique_seq_counts.rb:3:in `<main>'

    shopt -s extglob
    for f in ./Zipped/*; do
    gzcat $f | ./reformat_sequence_data.rb > $f.txt
    file=$(basename $f)
    python ./reformatsequence/$file.txt  ./qualitymask/$file.seq 20 1
    ./unique_seq_counts.rb ./qualitymask/$fileseq > ./uniquecounts/$file.counts.txt
    cut -f1 ./uniquecounts/$file.counts.txt > ./uniqueseq/$file.uniq.seq
    ./bowtie-1.1.0/bowtie -r -m1 -v1  ./bowtie-1.1.0/indexes/hg19 ./uniqueseq/$file.uniq.seq > ./alignedunique/$file.bowtie.txt

    Open in new window

    LVL 11

    Expert Comment

    Hi sebastizz,

    Regarding your post #40237238:

    > so does
    > for file in filename.*
    > mean for every file in the directory I specify as that would be ideal

    Not quite.  It means every file starting with 'filename.'.  Change the word 'filename' to anything you like, or change the template to match the files you want to match (e.g. '*.gz' to match all files with a 'gz' extension).
    LVL 48

    Expert Comment

    if you want all files do

    for file in *

    If you have subdirectories, then change to

    for file in $(find . -maxdepth 1 -type f)

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Join & Write a Comment

    If you other experts are anything like me you are always looking into and testing out new features. While I was doing some research one day I ran across an app that I installed on my Mac and used as a security system. Mac OS X: SecureHome uses your …
    Background Still having to process all these year-end "csv" files received from all these sources (including Government entities), sometimes we have the need to examine the contents due to data error, etc... As a "Unix" shop, our only readily …
    Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
    Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

    729 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now