Solved

Applying Logic To A Subset of File

Posted on 2008-10-01
3
306 Views
Last Modified: 2013-12-26
I have a script that searches a large file, and does arithmetic depending on what it finds.

Its a bit script but the bit of interest is:

for FILE in `ls $FILEMASK`
do
  while read line
  do
    if [ $1 = "NUMBER" ]
    then
      TOTAL=`expr $TOTAL + $2`
    fi
  done < $FILE
echo "Total is $TOTAL"
done

This works on all but the very largest of files.

To make it work for even the largest files, I want to egrep the file, and apply the locgic to the result of the egrep.

I guess I could egrep to a temporary file and change the filemask to process just that file, but is there a way to do it on the fly?
0
Comment
Question by:stummj
3 Comments
 
LVL 14

Accepted Solution

by:
sjm_ee earned 500 total points
ID: 22616275
for FILE in `ls $FILEMASK`
do
  egrep PATTERN $FILE | while read line
  do
    if [ $1 = "NUMBER" ]
    then
      TOTAL=`expr $TOTAL + $2`
    fi
  done
echo "Total is $TOTAL"
done
0
 

Author Closing Comment

by:stummj
ID: 31502033
Great. Thanks very much.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 22619259
You'll find that a while loop and expr will be very slow on large files.  *Much*  quicker to do
for file in $FILEMASK
do
  awk '/PATTERN/ && $1=="NUMBER" {t+=$2} END {print "Total is", t}' $file
done

Open in new window

0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Mysql command line display examples 11 84
Sed question 2 101
Debugging a shell script 3 31
sqlldr issue 1 15
Utilizing an array to gracefully append to a list of EmailAddresses
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question