We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you two Citrix podcasts. Learn about 2020 trends and get answers to your biggest Citrix questions!Listen Now

x

Bash shell script - batch process multiple files by command line

cybrthug
cybrthug asked
on
Medium Priority
3,448 Views
Last Modified: 2008-03-17
I have a bash script that works great when i process a single file by command line. I use

check 234234.txt

and it processes it great. I have a large directory full of .txt files that I want the script to process one by one and append all results to one single file for mysql import. I've tried doing check *.txt but it will only do the first file in the directory. Any ideas? Here is my current partial script.

if test $# -lt 1; then

        echo "usage: check <filename>";
        exit 1
fi


Thanks in advance!
Comment
Watch Question

Top Expert 2005

Commented:
You can iterate through args like

while [ -n "$1" ]; do
    echo $1;
    # do more job on $1
    shift
done

Author

Commented:
Hmm i tried working with this, it seems to see all the .txt files but it only processes the first one still, it does echo out each correct filename, but starts to reprocess the first .txt file in the directory after each echo. One thing that I do is append the filename to the end of each line when the script processes it, it uses the filename as an ID so i know where the line came from.
Top Expert 2005

Commented:
Can You show us the script itself, there's propably somewhere saved filename, so it uses same in each cycle.

Author

Commented:
Here she is:

#!/bin/bash

FILE="$1"
stringZ=$FILE
FILEA=`echo ${stringZ:(+1)}`
FILEB=`echo ${FILEA:0:5}`

if test $# -lt 1; then
        echo "usage: tabcon <filename>";
        exit 1
fi


while [ -n "$1" ]; do
    echo $1;
    shift

#read the file

while read DATA ; do

  #parse the leading line
  PITEM=`echo "$DATA" | cut -d ' ' -f1`
  PID=`echo "$DATA" | cut -d ' ' -f3`
  QUANTITY=`echo "$DATA" | cut -d ' ' -f4`

  DATA=`echo "$DATA" | cut -d ' ' -f11-`  # -f5-

  while [ "$QUANTITY" = "" ]; do
    QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
  done
  while [ 1 ]; do
  GREPCHECK=`echo "$QUANTITY" | grep ","`
  if [ -z "$GREPCHECK" ]; then
    QUANTITYFIX=`echo "$QUANTITY" | cut -d '.' -f1`
  else
    QUANTITYFIX=`echo "${QUANTITY//,/}" | cut -d '.' -f1`
  fi
  expr $QUANTITYFIX + 0 >/dev/null 2>&1
  if [ $? -ne 0 ]; then
    PID="$PID $QUANTITY"

    QUANTITY=`echo "$DATA" | cut -d ' ' -f2`

    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$QUANTITY" = "" ]; do
      QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
  else
 break
  fi
  done
  UM=`echo "$DATA" | cut -d ' ' -f4-`

#read the next line
  read DATA

  #test for a blank line
  while [ "$DATA" != "" ]; do
    #parse the item
    CID=`echo "$DATA" | cut -d ' ' -f1`
    CNTR=`echo "$DATA" | cut -d ' ' -f2`
    DATA=`echo "$DATA" | cut -d ' ' -f3-`
    while [ "$CNTR" = " " ]; do
      CNTR=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
    QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$QUANTITY" = " " ]; do
      QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
  while [ 1 ]; do

  GREPCHECK=`echo "$QUANTITY" | grep ","`
  if [ -z "$GREPCHECK" ]; then
    QUANTITYFIX=`echo "$QUANTITY" | cut -d '.' -f1`
  else
    QUANTITYFIX=`echo "${QUANTITY//,/}" | cut -d '.' -f1`
  fi
  expr $QUANTITYFIX + 1 >/dev/null 2>&1

  if [ $? -ne 0 ]; then
  CNTR="$CNTR $QUANTITY"
  QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
  DATA=`echo "$DATA" | cut -d ' ' -f2-`
  while [ "$QUANTITY" = "" ]; do
    QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
  done
else
break
fi
done
    CHKPR=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$CHKPR" = "" ]; do
      CHKPR=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
    TOTAL=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$TOTAL" = "" ]; do
      TOTAL=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
   echo "$CID;$PITEM;$QUANTITY;$UM;$CHKPR;$FILEB"

 #read the next line
    read DATA
  done

  #blank line
#  echo ""

done < "$FILE"

#done

done

exit 0
Top Expert 2005
Commented:
> FILE="$1"
> stringZ=$FILE
> FILEA=`echo ${stringZ:(+1)}`
> FILEB=`echo ${FILEA:0:5}`
here You set FILE variable, which is used then in all cycles. Right? Move this line to after the 'while [ -n "$1" ]; do'

also, 'shift' should be last statement in the 'while [ -n "$1" ]; do' loop, or at least should be after FILE names initializations. The 'shift' statement renames parameters $2 -> $1, $3 -> $2 and so on... so You always use $1 ;)

Is it clear enought?

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts

Author

Commented:
Simple enough, thank you so much, worked like a charm, just needed placement.
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.