Remove attributes from Jason from Linux

@enthuguy

#!/bin/bash
# Base version 0.0
for SRCFILE in "new_task_expected.json" "new_task_original.json"
do
	# Better use "$SRCFILE" instead of using $SRCFILE since source file names may have space.
	if [ -f "$SRCFILE" ]
	then
		if [ -r "$SRCFILE" ]
		then
			# If $SRCFILE having sourceContainer
			grep -l sourceContainer "$SRCFILE" >/dev/null 2>&1
			if [ 0 -eq $? ]
			then
				# $? used to have value 0
				if [ ! -f "$SRCFILE".OriginalFile.json ]
				then
					# Search sourceContainer and replace with murugesandins
					# /g => globally => all occurences
					# Take a backup of original file =>  $SRCFILE.OriginalFile.json
					echo sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
					sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
					# Update "$SRCFILE"
				else
					echo "Not using sed since $SRCFILE.OriginalFile.json file is present"
					echo "Update this script to replace .OriginalFile.json to other_text_uniqname.json"
				fi
			else
				echo "$SRCFILE not having sourceContainer"
			fi
		else
			echo "$SRCFILE file not having read permission."
		fi
	else
		echo "$SRCFILE file not found"
		echo "ls \"$SRCFILE\""
		ls "$SRCFILE"
	fi
done

Select allOpen in new window

sample output here:

$ ./29177755.sh
sed -i.OriginalFile.json s/sourceContainer/murugesandins/g; new_task_expected.json
sed -i.OriginalFile.json s/sourceContainer/murugesandins/g; new_task_original.json
$ grep murugesandins *.json
new_task_expected.json:                        "murugesandins": "simple-app"
new_task_original.json:                        "murugesandins": "simple-app"
$ grep sourceContainer *.json
new_task_expected.json.OriginalFile.json:                        "sourceContainer": "simple-app"
new_task_original.json.OriginalFile.json:                        "sourceContainer": "simple-app"

Select allOpen in new window

sample sed tutorial
https://www.tutorialspoint.com/sed/index.htm

Thanks again Murugasen for your time and help...to achieve this.

Thanks Louis, recently started using jq along with AWS CLI. its very powerful.

I will test both solutions and update back

Hi Murugesan,

A couple of comments about your proposed solution:

1. enthuguy has provided the original file (sample input) and expected file (sample output).  Why do you have a loop which seems to be processing both of them?  I'm referring to this loop:
    for SRCFILE in "new_task_expected.json" "new_task_original.json"
You should be processing the original input file to produce the expected output file, right?

2. The replacement you made seems to be a simple string replacement, i.e.:
    sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
If you compare the sample input and output files provided (with diff or whatever), you'll see the differences are not that simple.  So how is your method supposed to convert the sample input file into the expected output file?

I won't bother to provide an alternative solution, because doing such a task with standard Linux commands would probably be quite complex, and Louis's solution by installing the jq utility looks ideal, especially since enthuguy seems to have jq installed already.

Hi Murugesan, thx for your help and effort.

As we all would agree to tel2, even if my source input file changes, there would be a decent change has to be done to our script. jq on the other hand might need only minor tweak.

Going to try both script and jq solution provided today...will update today :)

Thanks again all.

Hi enthuguy,

It's great to have the sample input and output data.
However, a detailed description of what needs to change would also be useful.
Saying that you "would like to remove few attributes (block of lines) from a linux system" seems to be a bit (or even a byte) lacking, when I look at all the differences between the 2 files.  Here's the differences as reported by the "diff" command:

diff new_task_original.json new_task_expected.json
2,4c2
<     "taskDefinition": {
<         "status": "ACTIVE",
<         "family": "web-app-ecs-demo-app",
---
>         "family": "sunday-ecs-starttask-app",
6,22d3
<         "requiresAttributes": [
<             {
<                 "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
<             }
<         ],
<         "compatibilities": [
<             "EC2"
<         ],
30d10
<         "taskDefinitionArn": "arn:aws:ecs:ap-southeast-2:919280540730:task-definition/web-app-ecs-demo-app:3",
110,112c90
<         ],
<         "revision": 3
<     }
---
>         ]

Select allOpen in new window

Please correct me if I've missed something, but that doesn't look like just the removal of some attributes.  For starters, the input file has:  

        "family": "web-app-ecs-demo-app",

Select allOpen in new window

and the output file has:

        "family": "sunday-ecs-starttask-app",

Select allOpen in new window

which looks more like a change of a value than a removal of an attribute.

We need the detailed description so we can make this work for whatever input data you provide, otherwise you might end up with a solution which works perfectly for your sample data, but fails with some other data.

Are you with me?

@enthuguy

welcome
>> if my source input file changes, there would be a decent change has to be done to our script.
Hence posted the same which I am using a script for last 6 years related to work/money/KT/donations.

>> will update today
Thank you for handing svn/git/TFS/SMTP related actions to close current query at your location
(including accepted/assisted query at experts exchange).

Hi Murugesan,                     

>> if my source input file changes, there would be a decent change has to be done to our script.

Hence posted the same which I am using a script for last 6 years related to work/money/KT/donations.                 

I think you're missing his point...and my points.

Please read my 2 points about your solution, here:
  https://www.experts-exchange.com/questions/29177755/Remove-attributes-from-Jason-from-Linux.html#a43060972 
then respond to them.

And if you're still convinced that your solution will work, then please prove it by providing a version of it which takes the new_task_original.json file as input, and produces output identical to the new_task_expected.json file.  You'll find both those files attached to enthuguy's original post.

Thank you for handing svn/git/TFS/SMTP related actions to close current query at your location (including accepted/assisted query at experts exchange).

Comments like this are so cryptic that I doubt anyone reading this is going to understand you.  Communication is for the purpose of helping others understand things, so keep it simple enough to be understood, or you will just confuse people.  What is the relevance of SVN, GIT, TFS and SMTP to this question?

closing my comment here
zyxwvutsrqponmlkjihgfedcba

Thanks Louis LIETAER, that worked perfectly.

aws ecs describe-task-definition --task-definition ${latestTaskDef} | jq 'del(.taskDefinition.requiresAttributes)' | jq 'del(.taskDefinition.status)' | jq 'del(.taskDefinition.taskDefinitionArn)'  | jq 'del(.taskDefinition.revision)'  | jq 'del(.taskDefinition.compatibilities)' | grep -vwE "(taskDefinition)" | head -n -1 >  ${taskDefNewName}.json

Good to hear you worked it out using Louis's general suggestion, enthugy.

Can you show us the output you got from that command, please?
Is it really the same as your new_task_expected.json file?
If so, what part of your command resulted in this line (which was not in new_task_original.json) being added?:

    "family": "sunday-ecs-starttask-app",

Select allOpen in new window

Hi tel2, to replace the value I had to use sed command on top of it. (I had to quickly remove some string for security)
with my little knowledge. I had to do like this...

profileNum=$(grep -n 'PROFILE' ${taskDefNewName}.json | cut -d : -f 1) # get line num
profileNum=$((profileNum+1)).   # Increment to get profile value
sed -i '/sit1,aws/d' ${taskDefNewName}.json
eval $(echo "sed -i '${profileNum}i        "'"value"'": "'"sit1,aws,sftp,xxxxxx"'"' ${taskDefNewName}.json")
sed -i "s/${taskDefPrefix}/${taskDefNewPrefix}/g" ${taskDefNewName}.json    # update family name and contianer name

Select allOpen in new window

Are you sure it can't do the job in a simpler way with jq?  (I don't know jq, BTW.)

Assuming you can't...
Can you tell me what the dot at the end of this line is for, please?  [Update: This is a bit academic now, but I'd like to understand.]

profileNum=$((profileNum+1)).

Select allOpen in new window

I would have thought that dot would create an error when it ended up the sed command.  It gives me an error.  Does it work for you?

A shorter alternative in bash is to do this for incrementing a variable by 1:

(( profileNum++ ))

Select allOpen in new window

But sed can insert a line after it matches "PROFILE" if you like, using "a" instead of "i", something like this:

  sed -i '/PROFILE/a              ...etc...' $taskDefNewName.json   # No need for {} around variable names in most cases

Select allOpen in new window

Then you don't need to grep or increment profileNum at all.

But should "PROFILE" really be in UPPER case?  It isn't in the sample data you provided.
But I think there are simpler ways to do all this, but I'd need to better understand what you're currently doing first.  Namely:
What is in $taskDefPrefix and $taskDefNewPrefix?  [Update: Please give me some sample contents of those 2 variables.]
What is your overall strategy?  Is it this:
Line 1: Find the line that "PROFILE" is on.
Line 2: Increment that to get the line that the value is on.
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws".
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE".
Line 5: Replace (update) the family value.  How is the container name updated?

Hey tel2, thx very much for reviewing my script and logic. I didnt try sed a, yes that out save two line of my script :)

Yes, thats correct...below is the logic.
Line 3: in the entire file we will have  "sit1,aws" only once. So safe to use sed d
Line 5: I predefine the family name. e.g prefix is defined but suffix is dynamic. So i capture the suffix as an argument to this script.

Line 1: Find the line that "PROFILE" is on.
Line 2: Increment that to get the line that the value is on.
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws".
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE".
Line 5: Replace (update) the family value.  How is the container name updated?

Thanks for that info, enthugy,

I think we can improve this further.  If you want that, please answer the remaining outstanding questions, (which I have now underlined to you can easily spot them), from my previous post.  If askers answer all the questions experts ask, the first time they ask them, experts are more likely to be willing to spend their (unpaid) time on helping, and should be more equipped to give good answers first time, instead of having to guess.

Also, please answer these:
a) Please provide your updated code now that you've removed those 2 lines, etc.
b) Is it true that those 5 (now 3) lines are simply meant to?:
  i) Replace the value in the line immediately under the line which contains "PROFILE"?
 ii) Replace the family value.
c) How does this code fit in with your jq code?

Hi Tel2, thanks again, if we can improve, that would be a great help and a learning experience for me :)

Forgot to answer this "Are you sure it can't do the job in a simpler way with jq?" 
Honestly, I don't know, but will do some check.if jq can handle that too. that would be the cleaner way :)

a) Sure, I will provide the script shortly.
b) Yes, replacing the line would be good in my case, So we have full control. (what if the value already existing, then we might append and introduce duplicate? or we have to perform additional logic to check for the string we want to append "xxxxxx" already exist, then do not append, else get the current value and the append  "xxxxxx" ?
c) Not sure, I get this right. But right now jq is used only to delete the attribute from json. And sed is used to replace values.

Thanks again

Hi enthuguy,

Forgot to answer this " Are you sure it can't do the job in a simpler way with jq?"  
Honestly, I don't know, but will do some check.if jq can handle that too. that would be the cleaner way :)

That's the 1st of the 6 questions I underlined in that post.  Please answer the remaining 5.

:)
have already answered the very next comment. may be I did not explain well....sorry

Line 1: Find the line that "PROFILE" is on. -- Yes
Line 2: Increment that to get the line that the value is on. - Yes
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws". - Yes
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE". -Yes
Line 5: Replace (update) the family value.  How is the container name updated? by get few values as an argument to this script.

Those are not underlined questions in that post.  Can you see the 6 underlined questions in that post?  You've only answered the 1st now.

Sorry.  Correction - you've now answered the 1st & 6th question.
Here's my 6th question:
    "How is the container name updated?"
And your answer was:
    "by get few values as an argument to this script"
What I mean is, what part of your code is doing that?  And what container name are you referring to?

After some experimentation, I see values could be changed in your original file with this kind of jq syntax:

jq '.taskDefinition.family = "sunday-ecs-starttask-app" | .taskDefinition.containerDefinitions[].environment[].value = "sit1,aws,sftp,xxxxxx"' new_task_original.json >new_task_modified.json

Select allOpen in new window

Update: See how I've only called jq once, and used jq's pipe ("|") function to pass the output of one filter to the next.

thats great, if we can achieve using jq :)

My complete and complicated script  below :)

I also made a mistake in line. which I hardcoded for sit1.

sed -i '/sit1,aws/d' ${taskDefNewName}.json

Select allOpen in new window

 But this should be taken from

targetEnv=sit1

Select allOpen in new window

targetEnv=sit1
taskDefPrefix=docgen_postprocessor
ecsClusterPrefix=docgen-ms

taskDefPrefix=${base_ecs_task}
ecsCluster=${ecs_cluster}-${targetEnv}
taskDefPPName=${taskDefPrefix}_${targetEnv}
taskDefNewPrefix=${taskDefPrefix}_ondemand
taskDefNewName=${taskDefNewPrefix}_${targetEnv}

echo "Getting latest task definition for ${taskDefPPName}"
latestTaskDef=$(aws ecs list-task-definitions --region ap-southeast-2 | jq --raw-output ".taskDefinitionArns[]" | grep ${taskDefPPName} | sort -r -n -t ':' -k7 | head -n 1 | cut -d '/' -f 2)
echo "Found latest task definition ${latestTaskDef}"


echo "Getting latest task definition for ${taskDefPPName}"
aws ecs describe-task-definition --task-definition ${latestTaskDef} | jq 'del(.taskDefinition.requiresAttributes)' | jq 'del(.taskDefinition.status)' | jq 'del(.taskDefinition.taskDefinitionArn)'  | jq 'del(.taskDefinition.revision)'  | jq 'del(.taskDefinition.compatibilities)' | grep -vwE "(taskDefinition)" | head -n -1 >  ${taskDefNewName}.json


echo "Finally, rename the value of family in task definition ${taskDefNewName}.json"
profileNum=$(grep -n 'DOCGEN_POSTPROCESSOR_PROFILE' ${taskDefNewName}.json | cut -d : -f 1)
profileNum=$((profileNum+1))
sed -i '/sit1,aws/d' ${taskDefNewName}.json
eval $(echo "sed -i '${profileNum}i        "'"value"'": "'"sit1,aws,sftp,ondemand"'"' ${taskDefNewName}.json")
sed -i "s/${taskDefPrefix}/${taskDefNewPrefix}/g" ${taskDefNewName}.json    # update family name and contianer name


echo "create task definition..."
aws ecs register-task-definition --cli-input-json file://${taskDefNewName}.json


# Create Rule
aws events put-rule --schedule-expression "cron(0/55 * * * ? *)" --name ${taskDefNewName} --region ap-southeast-2
ruleARN=$(aws events describe-rule --name ${taskDefNewName} --query 'Arn' --output text)
ecsClusterARN=$(aws ecs describe-clusters  --clusters  ${ecsCluster} --query 'clusters[].clusterArn' --output text)
ecsEventRoleARN=$(aws iam list-roles --region ap-southeast-2 | jq --raw-output ".Roles[].Arn" | grep ecsEventsRole)
latestNewTaskDef=$(aws ecs list-task-definitions --region ap-southeast-2 | jq --raw-output ".taskDefinitionArns[]" | grep ${taskDefNewName} | sort -r -n -t ':' -k7 | head -n 1)


echo "Creating schedule task ${taskDefNewName} in cluster ${ecsCluster}."
aws events put-targets --rule ${taskDefNewName} --region ap-southeast-2 --targets "Id"="${taskDefNewName}","Arn"="${ecsClusterARN}","RoleArn"="${ecsEventRoleARN}","EcsParameters"="{"TaskDefinitionArn"="${latestNewTaskDef}","TaskCount"=1,"NetworkConfiguration"="{"awsvpcConfiguration"="{"Subnets"=["subnet-03c6ceb6de1d270a5,subnet-03deec33793694d61,subnet-00c1a58442ad5a0bf"],"SecurityGroups"="sg-009f7a78f99e1474d"}}}"

Select allOpen in new window

Thanks enthuguy.

Before we get into that, please answer the 4 outstandingunderlined questions from the post I've been referring to above.

HI Tel2, are you referring below, I answered it already saying "Yes"
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line w :)ith "PROFILE". -Yes

Your answers are not clear to me, enthuguy.

Please quote each of the 6 underlined questions I asked in that post, and paste the answers you have provided below each question, for clarity.

There is probably no need to answer anything on that post which is not underlined and does not end it a "?".

You have already answered the 1st one (about doing it in jq), and you've attempted to answer the 6th one (about the container), but the answer wasn't really what I was after so I asked a follow-up question here.

If you still don't understand what I'm asking for, let me know what's not clear.