Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Bash shell script - batch process multiple files by command line

Posted on 2006-03-27
6
Medium Priority
?
3,348 Views
Last Modified: 2008-03-17
I have a bash script that works great when i process a single file by command line. I use

check 234234.txt

and it processes it great. I have a large directory full of .txt files that I want the script to process one by one and append all results to one single file for mysql import. I've tried doing check *.txt but it will only do the first file in the directory. Any ideas? Here is my current partial script.

if test $# -lt 1; then

        echo "usage: check <filename>";
        exit 1
fi


Thanks in advance!
0
Comment
Question by:cybrthug
  • 3
  • 3
6 Comments
 
LVL 43

Expert Comment

by:ravenpl
ID: 16302465
You can iterate through args like

while [ -n "$1" ]; do
    echo $1;
    # do more job on $1
    shift
done
0
 

Author Comment

by:cybrthug
ID: 16302585
Hmm i tried working with this, it seems to see all the .txt files but it only processes the first one still, it does echo out each correct filename, but starts to reprocess the first .txt file in the directory after each echo. One thing that I do is append the filename to the end of each line when the script processes it, it uses the filename as an ID so i know where the line came from.
0
 
LVL 43

Expert Comment

by:ravenpl
ID: 16302663
Can You show us the script itself, there's propably somewhere saved filename, so it uses same in each cycle.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 

Author Comment

by:cybrthug
ID: 16302761
Here she is:

#!/bin/bash

FILE="$1"
stringZ=$FILE
FILEA=`echo ${stringZ:(+1)}`
FILEB=`echo ${FILEA:0:5}`

if test $# -lt 1; then
        echo "usage: tabcon <filename>";
        exit 1
fi


while [ -n "$1" ]; do
    echo $1;
    shift

#read the file

while read DATA ; do

  #parse the leading line
  PITEM=`echo "$DATA" | cut -d ' ' -f1`
  PID=`echo "$DATA" | cut -d ' ' -f3`
  QUANTITY=`echo "$DATA" | cut -d ' ' -f4`

  DATA=`echo "$DATA" | cut -d ' ' -f11-`  # -f5-

  while [ "$QUANTITY" = "" ]; do
    QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
  done
  while [ 1 ]; do
  GREPCHECK=`echo "$QUANTITY" | grep ","`
  if [ -z "$GREPCHECK" ]; then
    QUANTITYFIX=`echo "$QUANTITY" | cut -d '.' -f1`
  else
    QUANTITYFIX=`echo "${QUANTITY//,/}" | cut -d '.' -f1`
  fi
  expr $QUANTITYFIX + 0 >/dev/null 2>&1
  if [ $? -ne 0 ]; then
    PID="$PID $QUANTITY"

    QUANTITY=`echo "$DATA" | cut -d ' ' -f2`

    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$QUANTITY" = "" ]; do
      QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
  else
 break
  fi
  done
  UM=`echo "$DATA" | cut -d ' ' -f4-`

#read the next line
  read DATA

  #test for a blank line
  while [ "$DATA" != "" ]; do
    #parse the item
    CID=`echo "$DATA" | cut -d ' ' -f1`
    CNTR=`echo "$DATA" | cut -d ' ' -f2`
    DATA=`echo "$DATA" | cut -d ' ' -f3-`
    while [ "$CNTR" = " " ]; do
      CNTR=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
    QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$QUANTITY" = " " ]; do
      QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
  while [ 1 ]; do

  GREPCHECK=`echo "$QUANTITY" | grep ","`
  if [ -z "$GREPCHECK" ]; then
    QUANTITYFIX=`echo "$QUANTITY" | cut -d '.' -f1`
  else
    QUANTITYFIX=`echo "${QUANTITY//,/}" | cut -d '.' -f1`
  fi
  expr $QUANTITYFIX + 1 >/dev/null 2>&1

  if [ $? -ne 0 ]; then
  CNTR="$CNTR $QUANTITY"
  QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
  DATA=`echo "$DATA" | cut -d ' ' -f2-`
  while [ "$QUANTITY" = "" ]; do
    QUANTITY=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
  done
else
break
fi
done
    CHKPR=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$CHKPR" = "" ]; do
      CHKPR=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
    TOTAL=`echo "$DATA" | cut -d ' ' -f1`
    DATA=`echo "$DATA" | cut -d ' ' -f2-`
    while [ "$TOTAL" = "" ]; do
      TOTAL=`echo "$DATA" | cut -d ' ' -f1`
      DATA=`echo "$DATA" | cut -d ' ' -f2-`
    done
   echo "$CID;$PITEM;$QUANTITY;$UM;$CHKPR;$FILEB"

 #read the next line
    read DATA
  done

  #blank line
#  echo ""

done < "$FILE"

#done

done

exit 0
0
 
LVL 43

Accepted Solution

by:
ravenpl earned 700 total points
ID: 16302880
> FILE="$1"
> stringZ=$FILE
> FILEA=`echo ${stringZ:(+1)}`
> FILEB=`echo ${FILEA:0:5}`
here You set FILE variable, which is used then in all cycles. Right? Move this line to after the 'while [ -n "$1" ]; do'

also, 'shift' should be last statement in the 'while [ -n "$1" ]; do' loop, or at least should be after FILE names initializations. The 'shift' statement renames parameters $2 -> $1, $3 -> $2 and so on... so You always use $1 ;)

Is it clear enought?
0
 

Author Comment

by:cybrthug
ID: 16303258
Simple enough, thank you so much, worked like a charm, just needed placement.
0

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

578 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question