We help IT Professionals succeed at work.

Remove attributes from Jason from Linux

enthuguy
enthuguy asked
on
134 Views
Last Modified: 2020-04-13
Hi experts,

Have a json file, would like to remove few attributes (block of lines) from a linux system. could you help me what is the best way to achieve this please?

Have attached both original file and expected file for your quick reference

thanks in advance
new_task_expected.json
new_task_original.json
Comment
Watch Question

MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
@enthuguy

#!/bin/bash
# Base version 0.0
for SRCFILE in "new_task_expected.json" "new_task_original.json"
do
	# Better use "$SRCFILE" instead of using $SRCFILE since source file names may have space.
	if [ -f "$SRCFILE" ]
	then
		if [ -r "$SRCFILE" ]
		then
			# If $SRCFILE having sourceContainer
			grep -l sourceContainer "$SRCFILE" >/dev/null 2>&1
			if [ 0 -eq $? ]
			then
				# $? used to have value 0
				if [ ! -f "$SRCFILE".OriginalFile.json ]
				then
					# Search sourceContainer and replace with murugesandins
					# /g => globally => all occurences
					# Take a backup of original file =>  $SRCFILE.OriginalFile.json
					echo sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
					sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
					# Update "$SRCFILE"
				else
					echo "Not using sed since $SRCFILE.OriginalFile.json file is present"
					echo "Update this script to replace .OriginalFile.json to other_text_uniqname.json"
				fi
			else
				echo "$SRCFILE not having sourceContainer"
			fi
		else
			echo "$SRCFILE file not having read permission."
		fi
	else
		echo "$SRCFILE file not found"
		echo "ls \"$SRCFILE\""
		ls "$SRCFILE"
	fi
done

Open in new window

sample output here:
$ ./29177755.sh
sed -i.OriginalFile.json s/sourceContainer/murugesandins/g; new_task_expected.json
sed -i.OriginalFile.json s/sourceContainer/murugesandins/g; new_task_original.json
$ grep murugesandins *.json
new_task_expected.json:                        "murugesandins": "simple-app"
new_task_original.json:                        "murugesandins": "simple-app"
$ grep sourceContainer *.json
new_task_expected.json.OriginalFile.json:                        "sourceContainer": "simple-app"
new_task_original.json.OriginalFile.json:                        "sourceContainer": "simple-app"

Open in new window

sample sed tutorial
https://www.tutorialspoint.com/sed/index.htm
System Infrastructure Architect
CERTIFIED EXPERT
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Thanks again Murugasen for your time and help...to achieve this.

Thanks Louis, recently started using jq along with AWS CLI. its very powerful.

I will test both solutions and update back
CERTIFIED EXPERT

Commented:
Hi Murugesan,

A couple of comments about your proposed solution:

1. enthuguy has provided the original file (sample input) and expected file (sample output).  Why do you have a loop which seems to be processing both of them?  I'm referring to this loop:
    for SRCFILE in "new_task_expected.json" "new_task_original.json"
You should be processing the original input file to produce the expected output file, right?

2. The replacement you made seems to be a simple string replacement, i.e.:
    sed -i.OriginalFile.json "s/sourceContainer/murugesandins/g;" "$SRCFILE"
If you compare the sample input and output files provided (with diff or whatever), you'll see the differences are not that simple.  So how is your method supposed to convert the sample input file into the expected output file?

I won't bother to provide an alternative solution, because doing such a task with standard Linux commands would probably be quite complex, and Louis's solution by installing the jq utility looks ideal, especially since enthuguy seems to have jq installed already.

Author

Commented:
Hi Murugesan, thx for your help and effort.

As we all would agree to tel2, even if my source input file changes, there would be a decent change has to be done to our script. jq on the other hand might need only minor tweak.

Going to try both script and jq solution provided today...will update today :)

Thanks again all.
CERTIFIED EXPERT

Commented:
Hi enthuguy,

It's great to have the sample input and output data.
However, a detailed description of what needs to change would also be useful.
Saying that you "would like to remove few attributes (block of lines) from a linux system" seems to be a bit (or even a byte) lacking, when I look at all the differences between the 2 files.  Here's the differences as reported by the "diff" command:
diff new_task_original.json new_task_expected.json
2,4c2
<     "taskDefinition": {
<         "status": "ACTIVE",
<         "family": "web-app-ecs-demo-app",
---
>         "family": "sunday-ecs-starttask-app",
6,22d3
<         "requiresAttributes": [
<             {
<                 "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
<             },
<             {
<                 "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
<             }
<         ],
<         "compatibilities": [
<             "EC2"
<         ],
30d10
<         "taskDefinitionArn": "arn:aws:ecs:ap-southeast-2:919280540730:task-definition/web-app-ecs-demo-app:3",
110,112c90
<         ],
<         "revision": 3
<     }
---
>         ]
Please correct me if I've missed something, but that doesn't look like just the removal of some attributes.  For starters, the input file has:  
        "family": "web-app-ecs-demo-app",
and the output file has:
        "family": "sunday-ecs-starttask-app",
which looks more like a change of a value than a removal of an attribute.

We need the detailed description so we can make this work for whatever input data you provide, otherwise you might end up with a solution which works perfectly for your sample data, but fails with some other data.

Are you with me?

MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
@enthuguy

welcome
>> if my source input file changes, there would be a decent change has to be done to our script.
Hence posted the same which I am using a script for last 6 years related to work/money/KT/donations.
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
>> will update today
Thank you for handing svn/git/TFS/SMTP related actions to close current query at your location
(including accepted/assisted query at experts exchange).
CERTIFIED EXPERT

Commented:
Hi Murugesan,                     
>> if my source input file changes, there would be a decent change has to be done to our script.
Hence posted the same which I am using a script for last 6 years related to work/money/KT/donations.                 
I think you're missing his point...and my points.

Please read my 2 points about your solution, here:
  https://www.experts-exchange.com/questions/29177755/Remove-attributes-from-Jason-from-Linux.html#a43060972 
then respond to them.

And if you're still convinced that your solution will work, then please prove it by providing a version of it which takes the new_task_original.json file as input, and produces output identical to the new_task_expected.json file.  You'll find both those files attached to enthuguy's original post.


Thank you for handing svn/git/TFS/SMTP related actions to close current query at your location (including accepted/assisted query at experts exchange).
Comments like this are so cryptic that I doubt anyone reading this is going to understand you.  Communication is for the purpose of helping others understand things, so keep it simple enough to be understood, or you will just confuse people.  What is the relevance of SVN, GIT, TFS and SMTP to this question?
MURUGESAN NApplication Development and Automation Management
CERTIFIED EXPERT

Commented:
closing my comment here
zyxwvutsrqponmlkjihgfedcba

Author

Commented:
Thanks  Louis LIETAER, that worked perfectly.

aws ecs describe-task-definition --task-definition ${latestTaskDef} | jq 'del(.taskDefinition.requiresAttributes)' | jq 'del(.taskDefinition.status)' | jq 'del(.taskDefinition.taskDefinitionArn)'  | jq 'del(.taskDefinition.revision)'  | jq 'del(.taskDefinition.compatibilities)' | grep -vwE "(taskDefinition)" | head -n -1 >  ${taskDefNewName}.json
CERTIFIED EXPERT

Commented:
Good to hear you worked it out using Louis's general suggestion, enthugy.

Can you show us the output you got from that command, please?
Is it really the same as your new_task_expected.json file?
If so, what part of your command resulted in this line (which was not in new_task_original.json) being added?:
    "family": "sunday-ecs-starttask-app",

Author

Commented:
Hi tel2, to replace the value I had to use sed command on top of it. (I had to quickly remove some string for security)
with my little knowledge. I had to do like this...


profileNum=$(grep -n 'PROFILE' ${taskDefNewName}.json | cut -d : -f 1) # get line num
profileNum=$((profileNum+1)).   # Increment to get profile value
sed -i '/sit1,aws/d' ${taskDefNewName}.json
eval $(echo "sed -i '${profileNum}i        "'"value"'": "'"sit1,aws,sftp,xxxxxx"'"' ${taskDefNewName}.json")
sed -i "s/${taskDefPrefix}/${taskDefNewPrefix}/g" ${taskDefNewName}.json    # update family name and contianer name


CERTIFIED EXPERT

Commented:
Are you sure it can't do the job in a simpler way with jq?  (I don't know jq, BTW.)

Assuming you can't...
Can you tell me what the dot at the end of this line is for, please?  [Update: This is a bit academic now, but I'd like to understand.]
profileNum=$((profileNum+1)).
I would have thought that dot would create an error when it ended up the sed command.  It gives me an error.   Does it work for you?

A shorter alternative in bash is to do this for incrementing a variable by 1:
(( profileNum++ ))
But sed can insert a line after it matches "PROFILE" if you like, using "a" instead of "i", something like this:
  sed -i '/PROFILE/a              ...etc...' $taskDefNewName.json   # No need for {} around variable names in most cases
Then you don't need to grep or increment profileNum at all.

But should "PROFILE" really be in UPPER case?  It isn't in the sample data you provided.
But I think there are simpler ways to do all this, but I'd need to better understand what you're currently doing first.  Namely:
What is in $taskDefPrefix and $taskDefNewPrefix?  [Update: Please give me some sample contents of those 2 variables.]
What is your overall strategy?  Is it this:
Line 1: Find the line that "PROFILE" is on.
Line 2: Increment that to get the line that the value is on.
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws".
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE".
Line 5: Replace (update) the family value.   How is the container name updated?

Author

Commented:
Hey tel2, thx very much for reviewing my script and logic. I didnt try sed a, yes that out save two line of my script :)

Yes, thats correct...below is the logic.
Line 3: in the entire file we will have  "sit1,aws" only once. So safe to use sed d
Line 5: I predefine the family name. e.g prefix is defined but suffix is dynamic. So i capture the suffix as an argument to this script.

Line 1: Find the line that "PROFILE" is on.
Line 2: Increment that to get the line that the value is on.
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws".
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE".
Line 5: Replace (update) the family value.  How is the container name updated?
CERTIFIED EXPERT

Commented:
Thanks for that info, enthugy,

I think we can improve this further.  If you want that, please answer the remaining outstanding questions, (which I have now underlined to you can easily spot them), from my previous post.  If askers answer all the questions experts ask, the first time they ask them, experts are more likely to be willing to spend their (unpaid) time on helping, and should be more equipped to give good answers first time, instead of having to guess.

Also, please answer these:
a) Please provide your updated code now that you've removed those 2 lines, etc.
b) Is it true that those 5 (now 3) lines are simply meant to?:
  i) Replace the value in the line immediately under the line which contains "PROFILE"?
 ii) Replace the family value.
c) How does this code fit in with your jq code?

Author

Commented:
Hi Tel2, thanks again, if we can improve, that would be a great help and a learning experience for me :)

Forgot to answer this " Are you sure it can't do the job in a simpler way with jq?" 
Honestly, I don't know, but will do some check.if jq can handle that too. that would be the cleaner way :)

a) Sure, I will provide the script shortly.
b) Yes, replacing the line would be good in my case, So we have full control. (what if the value already existing, then we might append and introduce duplicate? or we have to perform additional logic to check for the string we want to append "xxxxxx" already exist, then do not append, else get the current value and the append  "xxxxxx" ?
c) Not sure, I get this right. But right now jq is used only to delete the attribute from json. And sed is used to replace values.

Thanks again

CERTIFIED EXPERT

Commented:
Hi enthuguy,
Forgot to answer this " Are you sure it can't do the job in a simpler way with jq?"  
Honestly, I don't know, but will do some check.if jq can handle that too. that would be the cleaner way :)
That's the 1st of the 6 questions I underlined in that post.  Please answer the remaining 5.

Author

Commented:
:)
have already answered the very next comment. may be I did not explain well....sorry

Line 1: Find the line that "PROFILE" is on. -- Yes
Line 2: Increment that to get the line that the value is on. - Yes
Line 3: Try to delete that line by deleting any line in the entire file which contains "sit1,aws". - Yes
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line with "PROFILE". -Yes
Line 5: Replace (update) the family value.  How is the container name updated? by get few values as an argument to this script.
CERTIFIED EXPERT

Commented:
Those are not underlined questions in that post.  Can you see the 6 underlined questions in that post?  You've only answered the 1st now.
CERTIFIED EXPERT

Commented:
Sorry.  Correction - you've now answered the 1st & 6th question.
Here's my 6th question:
    "How is the container name updated?"
And your answer was:
    "by get few values as an argument to this script"
What I mean is, what part of your code is doing that?  And what container name are you referring to?
CERTIFIED EXPERT

Commented:
After some experimentation, I see values could be changed in your original file with this kind of jq syntax:
jq '.taskDefinition.family = "sunday-ecs-starttask-app" | .taskDefinition.containerDefinitions[].environment[].value = "sit1,aws,sftp,xxxxxx"' new_task_original.json >new_task_modified.json
Update: See how I've only called jq once, and used jq's pipe ("|") function to pass the output of one filter to the next.

Author

Commented:
thats great, if we can achieve using jq :)

My complete and complicated script  below :)

I also made a mistake in line. which I hardcoded for sit1.
sed -i '/sit1,aws/d' ${taskDefNewName}.json
 But this should be taken from
targetEnv=sit1

targetEnv=sit1
taskDefPrefix=docgen_postprocessor
ecsClusterPrefix=docgen-ms

taskDefPrefix=${base_ecs_task}
ecsCluster=${ecs_cluster}-${targetEnv}
taskDefPPName=${taskDefPrefix}_${targetEnv}
taskDefNewPrefix=${taskDefPrefix}_ondemand
taskDefNewName=${taskDefNewPrefix}_${targetEnv}

echo "Getting latest task definition for ${taskDefPPName}"
latestTaskDef=$(aws ecs list-task-definitions --region ap-southeast-2 | jq --raw-output ".taskDefinitionArns[]" | grep ${taskDefPPName} | sort -r -n -t ':' -k7 | head -n 1 | cut -d '/' -f 2)
echo "Found latest task definition ${latestTaskDef}"


echo "Getting latest task definition for ${taskDefPPName}"
aws ecs describe-task-definition --task-definition ${latestTaskDef} | jq 'del(.taskDefinition.requiresAttributes)' | jq 'del(.taskDefinition.status)' | jq 'del(.taskDefinition.taskDefinitionArn)'  | jq 'del(.taskDefinition.revision)'  | jq 'del(.taskDefinition.compatibilities)' | grep -vwE "(taskDefinition)" | head -n -1 >  ${taskDefNewName}.json


echo "Finally, rename the value of family in task definition ${taskDefNewName}.json"
profileNum=$(grep -n 'DOCGEN_POSTPROCESSOR_PROFILE' ${taskDefNewName}.json | cut -d : -f 1)
profileNum=$((profileNum+1))
sed -i '/sit1,aws/d' ${taskDefNewName}.json
eval $(echo "sed -i '${profileNum}i        "'"value"'": "'"sit1,aws,sftp,ondemand"'"' ${taskDefNewName}.json")
sed -i "s/${taskDefPrefix}/${taskDefNewPrefix}/g" ${taskDefNewName}.json    # update family name and contianer name


echo "create task definition..."
aws ecs register-task-definition --cli-input-json file://${taskDefNewName}.json


# Create Rule
aws events put-rule --schedule-expression "cron(0/55 * * * ? *)" --name ${taskDefNewName} --region ap-southeast-2
ruleARN=$(aws events describe-rule --name ${taskDefNewName} --query 'Arn' --output text)
ecsClusterARN=$(aws ecs describe-clusters  --clusters  ${ecsCluster} --query 'clusters[].clusterArn' --output text)
ecsEventRoleARN=$(aws iam list-roles --region ap-southeast-2 | jq --raw-output ".Roles[].Arn" | grep ecsEventsRole)
latestNewTaskDef=$(aws ecs list-task-definitions --region ap-southeast-2 | jq --raw-output ".taskDefinitionArns[]" | grep ${taskDefNewName} | sort -r -n -t ':' -k7 | head -n 1)


echo "Creating schedule task ${taskDefNewName} in cluster ${ecsCluster}."
aws events put-targets --rule ${taskDefNewName} --region ap-southeast-2 --targets "Id"="${taskDefNewName}","Arn"="${ecsClusterARN}","RoleArn"="${ecsEventRoleARN}","EcsParameters"="{"TaskDefinitionArn"="${latestNewTaskDef}","TaskCount"=1,"NetworkConfiguration"="{"awsvpcConfiguration"="{"Subnets"=["subnet-03c6ceb6de1d270a5,subnet-03deec33793694d61,subnet-00c1a58442ad5a0bf"],"SecurityGroups"="sg-009f7a78f99e1474d"}}}"


CERTIFIED EXPERT

Commented:
Thanks enthuguy.

Before we get into that, please answer the 4 outstandingunderlined questions from the post I've been referring to above.

Author

Commented:
HI Tel2, are you referring below, I answered it already saying "Yes"
Line 4: Insert a new line containing: "value": "sit1,aws,sftp,xxxxxx", below the line w :)ith "PROFILE". - Yes
CERTIFIED EXPERT

Commented:
Your answers are not clear to me, enthuguy.

Please quote each of the 6 underlined questions I asked in that post, and paste the answers you have provided below each question, for clarity.

There is probably no need to answer anything on that post which is not underlined and does not end it a "?".

You have already answered the 1st one (about doing it in jq), and you've attempted to answer the 6th one (about the container), but the answer wasn't really what I was after so I asked a follow-up question here.

If you still don't understand what I'm asking for, let me know what's not clear.
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.