Solved

Applying Logic To A Subset of File

Posted on 2008-10-01
3
307 Views
Last Modified: 2013-12-26
I have a script that searches a large file, and does arithmetic depending on what it finds.

Its a bit script but the bit of interest is:

for FILE in `ls $FILEMASK`
do
  while read line
  do
    if [ $1 = "NUMBER" ]
    then
      TOTAL=`expr $TOTAL + $2`
    fi
  done < $FILE
echo "Total is $TOTAL"
done

This works on all but the very largest of files.

To make it work for even the largest files, I want to egrep the file, and apply the locgic to the result of the egrep.

I guess I could egrep to a temporary file and change the filemask to process just that file, but is there a way to do it on the fly?
0
Comment
Question by:stummj
3 Comments
 
LVL 14

Accepted Solution

by:
sjm_ee earned 500 total points
ID: 22616275
for FILE in `ls $FILEMASK`
do
  egrep PATTERN $FILE | while read line
  do
    if [ $1 = "NUMBER" ]
    then
      TOTAL=`expr $TOTAL + $2`
    fi
  done
echo "Total is $TOTAL"
done
0
 

Author Closing Comment

by:stummj
ID: 31502033
Great. Thanks very much.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 22619259
You'll find that a while loop and expr will be very slow on large files.  *Much*  quicker to do
for file in $FILEMASK
do
  awk '/PATTERN/ && $1=="NUMBER" {t+=$2} END {print "Total is", t}' $file
done

Open in new window

0

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Background Still having to process all these year-end "csv" files received from all these sources (including Government entities), sometimes we have the need to examine the contents due to data error, etc... As a "Unix" shop, our only readily …
Utilizing an array to gracefully append to a list of EmailAddresses
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…

685 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question