Unix ksh: mass replace of multiple values

Hi Experts,

A tough one, I'm afraid.
I need files to be mass changed.
The input file consists of 4 space/tabe seperated values.
Looks like this:

"Current Name"      "New Name"      "NEW USER"      "NEW ID"
MMDX-DE-D-C0973-AMAPS      MMMM-DE-D-C0973-AMAPS      VNXT3299      3291600

The file 'to be changed' has its entries in this format:

 SCRIPTNAME "/  -job MMDX-DE-D-C0973-AMAPS -user BG060733 -i 05340301 -c C"

After the change it should look like this:

 SCRIPTNAME "/  -job MMDX-DE-D-C0973-AMAPS -user VNXT3299 -i 3291600 -c C"
What needs to happen is this:
- all instances of MMDX-DE-D-C0973-AMAPS need replaced  by MMMM-DE-D-C0973-AMAPS
- the "-user" parameter needs changed from BG060733 to VNXT3299
- the "-i" parameter needs changed from 05340301 to 3291600

The input file has about 1500 lines.
Hope you can help.
Many thanks in advance.
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Frank ContrepoisVP Technical SalesCommented:
sed -e 's/MMDX-DE-D-C0973-AMAPS/MMMM-DE-D-C0973-AMAPS/g' \
       -e 's/BG060733/VNXT3299/g' \
       -e 's/05340301/3291600/g'

Open in new window

Try it first on a copy of the file so you don't mess with the real file
WatnogAuthor Commented:
To have the job name replaced with the sed \g would be ok.
A global change on user and jobnumber won't work as the old values are not known.

So when the old jobname is found:
- replace with second value of input line
In the line below that occurence:
- replace -job value with 2nd value of input line
- replace -user value with the 3rd value of the input line
- replace -i value with 4th value of input line

This needs to be repeated for each input line.
I imagine the script could walk these lines:

while read LINE; do
Current=$(echo $LINE | awk '{ print $1 }' )
NewJob=$(echo $LINE | awk '{ print $2 }' )
NewUSer=$(echo $LINE | awk '{ print $3 }' )
NewI=$(echo $LINE | awk '{ print $4 }'  )
[sed to find occurence of $Current and replace with $NewJob]
[sed to go one line lower and change -job value with $NewJob, change -user value with $NewUser, change -i value with $NewI]
done < inputfile

WatnogAuthor Commented:
set -x
while read LINE; do
Current=$(echo $LINE | awk '{ print $1 }' )
NewJob=$(echo $LINE | awk '{ print $2 }' )
NewUSer=$(echo $LINE | awk '{ print $3 }' )
NewI=$(echo $LINE | awk '{ print $4 }'  )
sed 's/$Current/$NewJob/' file_to_change
sed '/$NewJob/{n; s/-job .* -user/-job $NewJob -user/}' file_to_change  
sed '/$NewJob/{n; s/-user .* -i/-user $NewUser -i/}' file_to_change
sed '/$NewJob/{n; s/-i .* -c/-i $NewI -c/}' file_to_change                         
done < input

The{n; ...} is meant to look in the line below found pattern.
Yet it get a cannot parse error.

C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

WatnogAuthor Commented:
Response is a bit meager... I take the blame.

If you could help me on this I can struggle on...
I get stuck on this loop:

while read LINE
      Current=$(echo $LINE | awk '{ print $1 }' )
      New=$(echo $LINE | awk '{ print $2 }' )
      NewUSer=$(echo $LINE | awk '{ print $3 }' )
      NewI=$(echo $LINE | awk '{ print $4 }'  )
      echo Current=$Current
      echo New=$New
      sed "s/$Current/$New/" file_to_be_changed > changed_file
done < input

Only the first line of the input file is processed.
Any idea?

If you are using "> changed_file", then every loop will overwrite the output file with a new version, so only the last replacement woudl have been kept.

I think awk might be a better tool here:


while read curr newv newuser newi
  awk 'BEGIN{st=0}
$0 ~ /^[^#]*#'${curr}'/{split($0,aa,"#");print aa[1] "#" "'${newv}'";st=1;next}
st == 1{print " SCRIPTNAME \"/  -job '${newv}' -user '${newuser}' -i '${newi}' -c C\"";st=0;next}
{print}' kshrep.dat > kshrep.tmp
  mv kshrep.tmp kshrep.dat
done < kshrep.in

Open in new window

The quotes here are a bit tricky - probably best to cut and paste, and just change the name of the input file (the kshrep.in) and the file to be changed (here kshrep.dat).  Also make sure that the temporary file (kshrep.tmp) doesn't exist before you run the script.

Note that I have used multiple variable names on the "while read" line - that automatically splits the input line into multiple words, and avoids the need for your echo and awk statements.
WatnogAuthor Commented:
Cheers, many thanks, give me some time to check into this.
WatnogAuthor Commented:
If I run this, the kshrep.dat is emptied of all values:

 SCRIPTNAME "/  -job  -user  -i  -c C"
 DESCRIPTION "just a description"

In the 'real' kshrep.dat not all entries are candidate for replacment.
Maybe I should select out just those, and put them in same order as kshrep.in?

Thanks again.
WatnogAuthor Commented:
I could easily check that out myself in fact, but I'm a bit out of time, so forgive me.
I'll be back Monday.
It runs through the awk process once for each input line (so might be a little slow!).  The orde of entries in the input file does not matter, and each time it only updates the entry matching the first field of the input line - all other simply pass through unchanged.

Are all of the input lines the same format?  If one is empty, or is a "heading" line, that will confse things.

Are all of the patterns in the kshrep.dat file the same format?  My code assumes that the text to be matched is on a line immediately after a single "#" character (like the example in the original question).  It then assumes that the "SCRIPTNAME" line is always exactly the same format - the script has the SCRIPTNAME, -job, -i and -c C hardcoded, and prints out the values from the input file to fill in the variable data.

Have a great weekend, and let's talk on Monday.
WatnogAuthor Commented:
Ok, after tweaking the input file (kshrep.dat) it works, and that's great.
There are 2 issues, of which the first is less important.

1. Not all "scriptname" lines are identical
Apart from:
      SCRIPTNAME "/  -job MMDX-DE-D-C0973-AMAPS -user BG060733 -i 05340301 -c C"
Some have:
      SCRIPTNAME "/  -job ALE-AP-O-INBND_IDEIL -user US039332 -i 14100000 -c C -nobdcwait -flag DISABLE_JOBLOG"
      SCRIPTNAME "/  -job PPPU-BE-D-NEUPL-LTP-2-0538 -user EC004251 -i 05063700 -c C -nobdcwait"

Those extra parameters always come at the end (after the 'C') and don't need to be changed.
If it would be possible to count this possibilty in, that would be great, else I can treat them seperately changing the:
      st == 1{print " SCRIPTNAME \"/  -job '${newv}' -user '${newuser}' -i '${newi}' -c C\"";st=0;next}
line accordingly. So in fact that is no big deal.

2. This one is a show stopper (I think).
The full job definition has other lines uder "scriptname", as below.
I left those out to keep it simple... :-~

       SCRIPTNAME "/  -job PPPU-BE-D-PROD-ORDER-AVCHK-0811 -user CAN01351 -i 16433201 -c C"
       STREAMLOGON maestro
       DESCRIPTION "Mass Availlability Check P0811"

I cannot stripe those off on the 'real' file, so this should be taken into account.
The format is always the same:
      WORKSTATION#JOBNAME           # line to change
      SCRIPTNAME                    # line to change
      STREAMLOGON                                #  no change      
      DESCRIPTION                                #  no change
      TASKTYPE                                #  no change
      RECOVERY                              #  no change      

The second problem should be fine - the awk script finds the JOBNAME line and sets a flag (st=1) to say its found it.  When awk reads the next line, and st is set to 1, it assumes it is the SCRIPTNAME line and process it, then sets st to 0.  Then when awk reads in the next line (the STREAMLOGON one in your example), st is not 1, so awk just prints out the line unchanged.  It does this for all lines which aren't the JOBNAME one and aren't the line immediately following it.  To test this, have just one line in your "kshrep.in" file, and verify that the only changed lines between the kshrep.dat before and after are the JOBNAME and SCRIIPTNAME ones.

For the first problem, we can just get the awk line to print out all fields at the end of the line.  Assuming that the format of the first part (up to and including the "-c") is fixed, then you could replace the "st == 1" line with:
st == 1{printf " SCRIPTNAME \"/  -job '${newv}' -user '${newuser}' -i '${newi}'";for (ii=9; ii<=NF; ii++) {printf " %s",$ii;}printf "\n";st=0;next}

Open in new window


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
WatnogAuthor Commented:
Cheers. Yes that does work.

Can the JOBNAME be made an exact match?
finds/replaces also

WatnogAuthor Commented:
Thank  you simon3270.
I'm very much helped with  your solution.
Yes, it can be an exact match:


    $0 ~ /^[^#]*#'${curr}'/{split($0,aa,"#")


    $0 ~ /^[^#]*#'${curr}'$/{split($0,aa,"#")

(note the extra $ just before the /{)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.