Do not use on any
shared computer
August 29, 2008 02:59pm pdt
 
[x]
Attachment Details

Korn shell READ command taking hours to run through simple loop; can this be improved?

Tags: read, ksh, shell, korn, file
Hello

I have developed a script designed to read multiple files and extract spcific text information and write that info to an xml file. However, it takes many many hours (>24 in test). Since all the files are contained locally, I suspect the script itself is at fault.Please have a look at this code and file sample and help me find a more effect method. Kep in mind this is an old system. The version of MKS is unsupported and ancient (still waiting for a exact version from our Admins).

The translation file is built using SQR, so there are lots of odd characters that will not paste correctly into this text box, but the point of the file is to have element markers like such:

90v652H|element_X|0|value_X
90v2044H|element_X_X|1|value_X_X
180v3320H|element_X_X_X|2|value_X_X_X
270v4596H|element_X_X_Y|2|value_X_X_Y
270v4596H|element_X_Y|1|value_X_Y

The logic is that the XML processing job will extract the test between the pipes "|" and use the number as a tree index and the last part of the marker as the element value:

90v652H|element_X|0|value_X
90v2044H|element_X_X|1|value_X_X
180v3320H|element_X_X_X|2|value_X_X_X
270v4596H|element_X_X_Y|2|value_X_X_Y
270v4596H|element_X_Y|1|value_X_Y

Becomes

<element_X>value_X
      <element_X_X>value_X_X
            <element_X_X_X>value_X_X_X</element_X_X_X>
            <element_X_X_Y>value_X_X_Y</element_X_X_Y>
      </element_X_X>
      <element_X_Y></element_X_Y>
</element_X>

Here is the code:
#######Begin Code#######
#!/bin/ksh

USAGE="Usage :: parser.ksh [file name]"

#  Set Jobname
#
export JOBNAME=printing_xml
export FEEDER_DIR=${JOBS_DIR}${JOBNAME}/
REPORT_NAME=$1
export LIS_FILE=${FEEDER_DIR}Report/${REPORT_NAME}.lis
export LOG_FILE=${FEEDER_DIR}Log/${REPORT_NAME}_xml.log

echo `date` >> $LOG_FILE

case $# in
  1) if [ -e ${LIS_FILE} ]
     then
        echo LIS located > $LOG_FILE
     else
        echo Could not find ${LIS_FILE} >> $LOG_FILE
     exit    
     fi
     ;;

  *) # wrong parameters
     echo "Usage error" >> $LOG_FILE
     print $USAGE
     exit
     ;;
esac

echo "Setting up header and xml file" >> $LOG_FILE

XML_FILE=${FEEDER_DIR}Report/filename.xml
exec 3> "$XML_FILE"
echo "<?xml version="\""1.0"\"" encoding="\""ISO-8859-1"\""?>" >&3

#set up array tracker for walking through elements
ELEMENT_INDEX=-1

echo "Processing files" >> $LOG_FILE

LIS_COUNTER=0

#go through each file
while  [ -e $LIS_FILE ]
do
    echo "Reading " $LIS_FILE >> $LOG_FILE
    #Go through each line looking for '|'
   
    while read curr_line
    do
       CODE_VALIDATE=`echo "$curr_line" | grep -c "|"`
       
       if [ $CODE_VALIDATE -eq 1 ]
       then
           #round out the numbers
           EVAL_ELEMENT_VALUE=`echo "$curr_line" | cut -f4 -d"|"`
           if echo $EVAL_ELEMENT_VALUE | egrep '^-{0,1}[0-9]*\.[0-9]+$' >/dev/null 2>&1
           then
               EVAL_ELEMENT_VALUE=$(printf "%3.2f" $EVAL_ELEMENT_VALUE)
           fi
           
           if echo $EVAL_ELEMENT_VALUE | egrep '\&' >/dev/null 2>&1
           then
               EVAL_ELEMENT_VALUE=`echo "$EVAL_ELEMENT_VALUE" | sed -e "s/&/and/g"`
           fi

           #place element in xml file according to their index
           ELEMENT=`echo "$curr_line" | cut -f2 -d"|"`
           ELEMENT_INDEX_TMP=`echo "$curr_line" | cut -f3 -d"|"`
           ELEMENT_VALUE=$EVAL_ELEMENT_VALUE
           
           if [ $ELEMENT_INDEX_TMP -gt $ELEMENT_INDEX ]
           then
               ELEMENT_INDEX=$ELEMENT_INDEX_TMP
               ARR_ELEMENTS[$ELEMENT_INDEX]="$ELEMENT"
               echo "<${ARR_ELEMENTS[$ELEMENT_INDEX]}>${ELEMENT_VALUE}" >&3
           elif [ $ELEMENT_INDEX_TMP -eq $ELEMENT_INDEX ]
           then
               echo "</${ARR_ELEMENTS[$ELEMENT_INDEX]}>" >&3
               ELEMENT_INDEX=$ELEMENT_INDEX_TMP
               ARR_ELEMENTS[$ELEMENT_INDEX]="$ELEMENT"
               echo "<${ARR_ELEMENTS[$ELEMENT_INDEX]}>${ELEMENT_VALUE}" >&3
           else
               while [ $ELEMENT_INDEX_TMP -lt $ELEMENT_INDEX ]
               do
                   echo "</${ARR_ELEMENTS[$ELEMENT_INDEX]}>" >&3
                   ELEMENT_INDEX=$ELEMENT_INDEX-1
               done
               echo "</${ARR_ELEMENTS[$ELEMENT_INDEX]}>" >&3
               ELEMENT_INDEX=$ELEMENT_INDEX_TMP
               ARR_ELEMENTS[$ELEMENT_INDEX]="$ELEMENT"
               echo "<${ARR_ELEMENTS[$ELEMENT_INDEX]}>${ELEMENT_VALUE}" >&3
           fi          
       fi
    done  < "$LIS_FILE"
   
    let "LIS_COUNTER=$LIS_COUNTER+1"
   
    if [ $LIS_COUNTER -gt 9 ]
    then
        LIS_FILE="${FEEDER_DIR}Report/${REPORT_NAME}_${LIS_COUNTER}.lis"
    else
        LIS_FILE="${FEEDER_DIR}Report/${REPORT_NAME}_0${LIS_COUNTER}.lis"
    fi
done

#clean up remaining tags
while [ $ELEMENT_INDEX -gt 0 ]
do
    echo "</${ARR_ELEMENTS[$ELEMENT_INDEX]}>" >&3
    ELEMENT_INDEX=$ELEMENT_INDEX-1
done
echo "</${ARR_ELEMENTS[$ELEMENT_INDEX]}>" >&3

#######END Code#######

Can someone help me optimize this job and reduce it's run time greatly?

Also, for a hero cookie, I noticed the code that puts the closing tags on the end of the document failed to print on a big run (10+ meg), this may be part of the multiple file logic, but some feedback on that would be appreciated too.
Start your free trial to view this solution
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

Question Stats
Zone: Programming
Question Asked By: AFSCOPS
Solution Provided By: ahoffmann
Participating Experts: 3
Solution Grade: A
Views: 141
Translate:
Loading Advertisement...
 
[+][-]Expert Comment by tfewster

Rank: Master

Expert Comment by tfewster:

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
[+][-]Expert Comment by Tintin

Rank: Wizard

Expert Comment by Tintin:

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
[+][-]Author Comment by AFSCOPS
Author Comment by AFSCOPS:

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
[+][-]Accepted Solution by ahoffmann

Rank: Wizard

Accepted Solution by ahoffmann:

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
[+][-]Author Comment by AFSCOPS
Author Comment by AFSCOPS:

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
[+][-]Expert Comment by ahoffmann

Rank: Wizard

Expert Comment by ahoffmann:

All comments and solutions are available to Premium Service Members only.

Start your 7-day free trial and see for yourself why Experts Exchange is the easiest and most proven technology resource in the world. Get Started

Already a member? Login to view this solution.

 
 
Loading Advertisement...
20080723-EE-VQP-34 / EE_QW_2_20070628