Avatar of enthuguy
enthuguyFlag for Australia asked on

Remove attributes from Jason from Linux

Hi experts,

Have a json file, would like to remove few attributes (block of lines) from a linux system. could you help me what is the best way to achieve this please?

Have attached both original file and expected file for your quick reference

thanks in advance
new_task_expected.json
new_task_original.json
Linux* BashShell ScriptingJSON

Avatar of undefined
Last Comment
tel2

8/22/2022 - Mon
MURUGESAN N

@enthuguy

#!/bin/bash
# Base version 0.0
for SRCFILE in "new_task_expected.json" "new_task_original.json"
do
	# Better use "$SRCFILE" instead of using $SRCFILE since source file names may have space.
	if [ -f "$SRCFILE" ]
	then
		if [ -r "$SRCFILE" ]
		then
			# If $SRCFILE having sourceContainer
			grep -l sourceContainer "$SRCFILE" >/dev/null 2>&1
			if [ 0 -eq $? ]
			then
				# $? used to have value 0
				if [ ! -f "$SRCFILE".OriginalFile.json ]
				then
					# Search sourceContainer and replace with murugesandins
					# /g => globally => all occurences
					# Take a backup of original file =>  $SRCFILE.OriginalFile.json
					echo sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
					sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
					# Update "$SRCFILE"
				else
					echo "Not using sed since $SRCFILE.OriginalFile.json file is present"
					echo "Update this script to replace .OriginalFile.json to other_text_uniqname.json"
				fi
			else
				echo "$SRCFILE not having sourceContainer"
			fi
		else
			echo "$SRCFILE file not having read permission."
		fi
	else
		echo "$SRCFILE file not found"
		echo "ls \"$SRCFILE\""
		ls "$SRCFILE"
	fi
done

Open in new window

sample output here:
$ ./29177755.sh
sed -i.OriginalFile.json s/sourceContainer/murugesandins/g; new_task_expected.json
sed -i.OriginalFile.json s/sourceContainer/murugesandins/g; new_task_original.json
$ grep murugesandins *.json
new_task_expected.json:                        "murugesandins": "simple-app"
new_task_original.json:                        "murugesandins": "simple-app"
$ grep sourceContainer *.json
new_task_expected.json.OriginalFile.json:                        "sourceContainer": "simple-app"
new_task_original.json.OriginalFile.json:                        "sourceContainer": "simple-app"

Open in new window

sample sed tutorial
https://www.tutorialspoint.com/sed/index.htm
ASKER CERTIFIED SOLUTION
Louis LIETAER

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
ASKER
enthuguy

Thanks again Murugasen for your time and help...to achieve this.

Thanks Louis, recently started using jq along with AWS CLI. its very powerful.

I will test both solutions and update back
tel2

Hi Murugesan,

A couple of comments about your proposed solution:

1. enthuguy has provided the original file (sample input) and expected file (sample output).  Why do you have a loop which seems to be processing both of them?  I'm referring to this loop:
    for SRCFILE in "new_task_expected.json" "new_task_original.json"
You should be processing the original input file to produce the expected output file, right?

2. The replacement you made seems to be a simple string replacement, i.e.:
    sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
If you compare the sample input and output files provided (with diff or whatever), you'll see the differences are not that simple.  So how is your method supposed to convert the sample input file into the expected output file?

I won't bother to provide an alternative solution, because doing such a task with standard Linux commands would probably be quite complex, and Louis's solution by installing the jq utility looks ideal, especially since enthuguy seems to have jq installed already.
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
ASKER
enthuguy

Hi Murugesan, thx for your help and effort.

As we all would agree to tel2, even if my source input file changes, there would be a decent change has to be done to our script. jq on the other hand might need only minor tweak.

Going to try both script and jq solution provided today...will update today :)

Thanks again all.
tel2

Hi enthuguy,

It's great to have the sample input and output data.
However, a detailed description of what needs to change would also be useful.
Saying that you "would like to remove few attributes (block of lines) from a linux system" seems to be a bit (or even a byte) lacking, when I look at all the differences between the 2 files.  Here's the differences as reported by the "diff" command:
diff new_task_original.json new_task_expected.json
2,4c2
<     "taskDefinition": {
<         "status": "ACTIVE",
<         "family": "web-app-ecs-demo-app",
---
>         "family": "sunday-ecs-starttask-app",
6,22d3
<         "requiresAttributes": [
<             {
<                 "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
<             }
<         ],
<         "compatibilities": [
<             "EC2"
<         ],
30d10
<         "taskDefinitionArn": "arn:aws:ecs:ap-southeast-2:919280540730:task-definition/web-app-ecs-demo-app:3",
110,112c90
<         ],
<         "revision": 3
<     }
---
>         ]

Open in new window

Please correct me if I've missed something, but that doesn't look like just the removal of some attributes.  For starters, the input file has:  
        "family": "web-app-ecs-demo-app",

Open in new window

and the output file has:
        "family": "sunday-ecs-starttask-app",

Open in new window

which looks more like a change of a value than a removal of an attribute.

We need the detailed description so we can make this work for whatever input data you provide, otherwise you might end up with a solution which works perfectly for your sample data, but fails with some other data.

Are you with me?

MURUGESAN N

@enthuguy

welcome
>> if my source input file changes, there would be a decent change has to be done to our script.
Hence posted the same which I am using a script for last 6 years related to work/money/KT/donations.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
MURUGESAN N

>> will update today
Thank you for handing svn/git/TFS/SMTP related actions to close current query at your location
(including accepted/assisted query at experts exchange).
tel2

Hi Murugesan,                     
>> if my source input file changes, there would be a decent change has to be done to our script.
Hence posted the same which I am using a script for last 6 years related to work/money/KT/donations.                 
I think you're missing his point...and my points.

Please read my 2 points about your solution, here:
  https://www.experts-exchange.com/questions/29177755/Remove-attributes-from-Jason-from-Linux.html#a43060972 
then respond to them.

And if you're still convinced that your solution will work, then please prove it by providing a version of it which takes the new_task_original.json file as input, and produces output identical to the new_task_expected.json file.  You'll find both those files attached to enthuguy's original post.


Thank you for handing svn/git/TFS/SMTP related actions to close current query at your location (including accepted/assisted query at experts exchange).
Comments like this are so cryptic that I doubt anyone reading this is going to understand you.  Communication is for the purpose of helping others understand things, so keep it simple enough to be understood, or you will just confuse people.  What is the relevance of SVN, GIT, TFS and SMTP to this question?
MURUGESAN N

closing my comment here
zyxwvutsrqponmlkjihgfedcba
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
ASKER
enthuguy

Thanks Louis LIETAER, that worked perfectly.

aws ecs describe-task-definition --task-definition ${latestTaskDef} | jq 'del(.taskDefinition.requiresAttributes)' | jq 'del(.taskDefinition.status)' | jq 'del(.taskDefinition.taskDefinitionArn)'  | jq 'del(.taskDefinition.revision)'  | jq 'del(.taskDefinition.compatibilities)' | grep -vwE "(taskDefinition)" | head -n -1 >  ${taskDefNewName}.json
tel2

Good to hear you worked it out using Louis's general suggestion, enthugy.

Can you show us the output you got from that command, please?
Is it really the same as your new_task_expected.json file?
If so, what part of your command resulted in this line (which was not in new_task_original.json) being added?:
    "family": "sunday-ecs-starttask-app",

Open in new window

ASKER
enthuguy

Hi tel2, to replace the value I had to use sed command on top of it. (I had to quickly remove some string for security)
with my little knowledge. I had to do like this...


profileNum=$(grep -n 'PROFILE' ${taskDefNewName}.json | cut -d : -f 1) # get line num
profileNum=$((profileNum+1)).   # Increment to get profile value
sed -i '/sit1,aws/d' ${taskDefNewName}.json
eval $(echo "sed -i '${profileNum}i        "'"value"'": "'"sit1,aws,sftp,xxxxxx"'"' ${taskDefNewName}.json")
sed -i "s/${taskDefPrefix}/${taskDefNewPrefix}/g" ${taskDefNewName}.json    # update family name and contianer name

Open in new window



Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
tel2

Are you sure it can't do the job in a simpler way with jq?  (I don't know jq, BTW.)

Assuming you can't...
Can you tell me what the dot at the end of this line is for, please?  [Update: This is a bit academic now, but I'd like to understand.]
profileNum=$((profileNum+1)).

Open in new window

I would have thought that dot would create an error when it ended up the sed command.  It gives me an error.  Does it work for you?

A shorter alternative in bash is to do this for incrementing a variable by 1:
(( profileNum++ ))

Open in new window

But sed can insert a line after it matches "PROFILE" if you like, using "a" instead of "i", something like this:
  sed -i '/PROFILE/a              ...etc...' $taskDefNewName.json   # No need for {} around variable names in most cases

Open in new window

Then you don't need to grep or increment profileNum at all.

But should "PROFILE" really be in UPPER case?  It isn't in the sample data you provided.
But I think there are simpler ways to do all this, but I'd need to better understand what you're currently doing first.  Namely:
What is in $taskDefPrefix and $taskDefNewPrefix?  [Update: Please give me some sample contents of those 2 variables.]
What is your overall strategy?  Is it this:
Line 1: Find the line that "PROFILE" is on.
Line 2: Increment that to get the line that the value is on.
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws".
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE".
Line 5: Replace (update) the family value.  How is the container name updated?
ASKER
enthuguy

Hey tel2, thx very much for reviewing my script and logic. I didnt try sed a, yes that out save two line of my script :)

Yes, thats correct...below is the logic.
Line 3: in the entire file we will have  "sit1,aws" only once. So safe to use sed d
Line 5: I predefine the family name. e.g prefix is defined but suffix is dynamic. So i capture the suffix as an argument to this script.

Line 1: Find the line that "PROFILE" is on.
Line 2: Increment that to get the line that the value is on.
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws".
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE".
Line 5: Replace (update) the family value.  How is the container name updated?
tel2

Thanks for that info, enthugy,

I think we can improve this further.  If you want that, please answer the remaining outstanding questions, (which I have now underlined to you can easily spot them), from my previous post.  If askers answer all the questions experts ask, the first time they ask them, experts are more likely to be willing to spend their (unpaid) time on helping, and should be more equipped to give good answers first time, instead of having to guess.

Also, please answer these:
a) Please provide your updated code now that you've removed those 2 lines, etc.
b) Is it true that those 5 (now 3) lines are simply meant to?:
  i) Replace the value in the line immediately under the line which contains "PROFILE"?
 ii) Replace the family value.
c) How does this code fit in with your jq code?
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
ASKER
enthuguy

Hi Tel2, thanks again, if we can improve, that would be a great help and a learning experience for me :)

Forgot to answer this "Are you sure it can't do the job in a simpler way with jq?" 
Honestly, I don't know, but will do some check.if jq can handle that too. that would be the cleaner way :)

a) Sure, I will provide the script shortly.
b) Yes, replacing the line would be good in my case, So we have full control. (what if the value already existing, then we might append and introduce duplicate? or we have to perform additional logic to check for the string we want to append "xxxxxx" already exist, then do not append, else get the current value and the append  "xxxxxx" ?
c) Not sure, I get this right. But right now jq is used only to delete the attribute from json. And sed is used to replace values.

Thanks again

tel2

Hi enthuguy,
Forgot to answer this " Are you sure it can't do the job in a simpler way with jq?"  
Honestly, I don't know, but will do some check.if jq can handle that too. that would be the cleaner way :)
That's the 1st of the 6 questions I underlined in that post.  Please answer the remaining 5.
ASKER
enthuguy

:)
have already answered the very next comment. may be I did not explain well....sorry

Line 1: Find the line that "PROFILE" is on. -- Yes
Line 2: Increment that to get the line that the value is on. - Yes
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws". - Yes
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE". -Yes
Line 5: Replace (update) the family value.  How is the container name updated? by get few values as an argument to this script.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
tel2

Those are not underlined questions in that post.  Can you see the 6 underlined questions in that post?  You've only answered the 1st now.
tel2

Sorry.  Correction - you've now answered the 1st & 6th question.
Here's my 6th question:
    "How is the container name updated?"
And your answer was:
    "by get few values as an argument to this script"
What I mean is, what part of your code is doing that?  And what container name are you referring to?
tel2

After some experimentation, I see values could be changed in your original file with this kind of jq syntax:
jq '.taskDefinition.family = "sunday-ecs-starttask-app" | .taskDefinition.containerDefinitions[].environment[].value = "sit1,aws,sftp,xxxxxx"' new_task_original.json >new_task_modified.json

Open in new window

Update: See how I've only called jq once, and used jq's pipe ("|") function to pass the output of one filter to the next.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
ASKER
enthuguy

thats great, if we can achieve using jq :)

My complete and complicated script  below :)

I also made a mistake in line. which I hardcoded for sit1.
sed -i '/sit1,aws/d' ${taskDefNewName}.json

Open in new window

 But this should be taken from
targetEnv=sit1

Open in new window


targetEnv=sit1
taskDefPrefix=docgen_postprocessor
ecsClusterPrefix=docgen-ms

taskDefPrefix=${base_ecs_task}
ecsCluster=${ecs_cluster}-${targetEnv}
taskDefPPName=${taskDefPrefix}_${targetEnv}
taskDefNewPrefix=${taskDefPrefix}_ondemand
taskDefNewName=${taskDefNewPrefix}_${targetEnv}

echo "Getting latest task definition for ${taskDefPPName}"
latestTaskDef=$(aws ecs list-task-definitions --region ap-southeast-2 | jq --raw-output ".taskDefinitionArns[]" | grep ${taskDefPPName} | sort -r -n -t ':' -k7 | head -n 1 | cut -d '/' -f 2)
echo "Found latest task definition ${latestTaskDef}"


echo "Getting latest task definition for ${taskDefPPName}"
aws ecs describe-task-definition --task-definition ${latestTaskDef} | jq 'del(.taskDefinition.requiresAttributes)' | jq 'del(.taskDefinition.status)' | jq 'del(.taskDefinition.taskDefinitionArn)'  | jq 'del(.taskDefinition.revision)'  | jq 'del(.taskDefinition.compatibilities)' | grep -vwE "(taskDefinition)" | head -n -1 >  ${taskDefNewName}.json


echo "Finally, rename the value of family in task definition ${taskDefNewName}.json"
profileNum=$(grep -n 'DOCGEN_POSTPROCESSOR_PROFILE' ${taskDefNewName}.json | cut -d : -f 1)
profileNum=$((profileNum+1))
sed -i '/sit1,aws/d' ${taskDefNewName}.json
eval $(echo "sed -i '${profileNum}i        "'"value"'": "'"sit1,aws,sftp,ondemand"'"' ${taskDefNewName}.json")
sed -i "s/${taskDefPrefix}/${taskDefNewPrefix}/g" ${taskDefNewName}.json    # update family name and contianer name


echo "create task definition..."
aws ecs register-task-definition --cli-input-json file://${taskDefNewName}.json


# Create Rule
aws events put-rule --schedule-expression "cron(0/55 * * * ? *)" --name ${taskDefNewName} --region ap-southeast-2
ruleARN=$(aws events describe-rule --name ${taskDefNewName} --query 'Arn' --output text)
ecsClusterARN=$(aws ecs describe-clusters  --clusters  ${ecsCluster} --query 'clusters[].clusterArn' --output text)
ecsEventRoleARN=$(aws iam list-roles --region ap-southeast-2 | jq --raw-output ".Roles[].Arn" | grep ecsEventsRole)
latestNewTaskDef=$(aws ecs list-task-definitions --region ap-southeast-2 | jq --raw-output ".taskDefinitionArns[]" | grep ${taskDefNewName} | sort -r -n -t ':' -k7 | head -n 1)


echo "Creating schedule task ${taskDefNewName} in cluster ${ecsCluster}."
aws events put-targets --rule ${taskDefNewName} --region ap-southeast-2 --targets "Id"="${taskDefNewName}","Arn"="${ecsClusterARN}","RoleArn"="${ecsEventRoleARN}","EcsParameters"="{"TaskDefinitionArn"="${latestNewTaskDef}","TaskCount"=1,"NetworkConfiguration"="{"awsvpcConfiguration"="{"Subnets"=["subnet-03c6ceb6de1d270a5,subnet-03deec33793694d61,subnet-00c1a58442ad5a0bf"],"SecurityGroups"="sg-009f7a78f99e1474d"}}}"

Open in new window


tel2

Thanks enthuguy.

Before we get into that, please answer the 4 outstandingunderlined questions from the post I've been referring to above.
ASKER
enthuguy

HI Tel2, are you referring below, I answered it already saying "Yes"
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line w :)ith "PROFILE". -Yes
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
tel2

Your answers are not clear to me, enthuguy.

Please quote each of the 6 underlined questions I asked in that post, and paste the answers you have provided below each question, for clarity.

There is probably no need to answer anything on that post which is not underlined and does not end it a "?".

You have already answered the 1st one (about doing it in jq), and you've attempted to answer the 6th one (about the container), but the answer wasn't really what I was after so I asked a follow-up question here.

If you still don't understand what I'm asking for, let me know what's not clear.