Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

Unix/Linux Help for the below command

Posted on 2011-09-30
9
461 Views
Last Modified: 2013-11-22
Hi,

Can anyone pleas let me know  how the below command works.

awk  '/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}' HD_sync.log.10|sed -ne 's/^[^/]*[/]\(IMD.*\)\/IMAGES\//\1/p'|sort |uniq
0
Comment
Question by:new_perl_user
  • 4
  • 3
  • 2
9 Comments
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 36891941
Hi, it is a 'one liner' that processed the file HD_sync.log.10 with awk and sed and then sorts the output leaving only the unique entries.

Here's a description of what is happening. More details can be given if can post a sample HD_sync.log file.

awk'/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}'
1. Perform this awk command - for lines that match pattern STARTED or FINISHED - check if pattern /IMAGES[/]$/ is on this line, if so output the 2nd field of that line

HD_sync.log.10
2. on this file (your input file)

|sed -ne 's/^[^/]*[/]\(IMD.*\)\/IMAGES\//\1/p'
3. search beginning (^) of each line for pattern [^/]*[/]\(IMD.*\)\/IMAGES\/ and print the first matching pattern

|sort
4. sort the output

|uniq
5. keep only unique lines

Is this enough explanation for you? The above is really not 'a command' but a series of commands on the input file. You can view the output steps in between by doing these separate steps:

awk  '/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}' HD_sync.log.10
awk  '/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}' HD_sync.log.10|sed -ne 's/^[^/]*[/]\(IMD.*\)\/IMAGES\//\1/p'
awk  '/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}' HD_sync.log.10|sed -ne 's/^[^/]*[/]\(IMD.*\)\/IMAGES\//\1/p'|sort
awk  '/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}' HD_sync.log.10|sed -ne 's/^[^/]*[/]\(IMD.*\)\/IMAGES\//\1/p'|sort |uniq
0
 

Author Comment

by:new_perl_user
ID: 36892020
Hi,
Thank you so much for the above explanation it is crystal clear. Also need a small help.  how can we tweak the above series of commands  to grab  data based on date(it should get yesterday's data from the file)  and write it to a log file.
0
 
LVL 38

Expert Comment

by:wesly_chen
ID: 36892663
> awk'/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}'
> 1. Perform this awk command - for lines that match pattern STARTED or FINISHED
No
In HD_sync.log.10, search the content between "STARTED" and "FINISHED", If any line contain "IMAGES/" or "IMAGES" at the end of line, then print the
second field of that line

> grab  data based on date(it should get yesterday
You need to provide the date format and the location of date in  a line, like
2011/09/29 or
2011-09-29
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 

Author Comment

by:new_perl_user
ID: 36892746
I can't hardcode the date. so is there a way to  tell it  to look for data sysdate -1
0
 
LVL 38

Expert Comment

by:wesly_chen
ID: 36892807
> can't hardcode the date.
You just need to look at the HD_sync.log.10 and post the date/time format in that file (more than one for comparison)
0
 

Author Comment

by:new_perl_user
ID: 36892847
the date format in the file is:'Mon Sep 26 05:10:05 ' and this log file  gets data every hour  365 days . So everday I need to run the above command throught script and grab the data for yesterday . will it be possible, if so  can you please let me know.

Thanks,
0
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 36893286
@wesley_chen
>> search the content between "STARTED" and "FINISHED"
You are absolutely correct, sorry about that.

About adding the date filter: I would add this:

grep "`date -d 'yesterday' +"%a %b %d"`"

then add the (somewhat modified) original line to this:

grep "`date -d 'yesterday' +"%a %b %d"`" | awk  '/STARTED/,/FINISHED/{if($0~/IMAGES[/]$/){print $2}}' | sed -ne 's/^[^/]*[/]\(IMD.*\)\/IMAGES\//\1/p' | sort | uniq

Open in new window


The 'grep' line will filter the lines matching 'yesterday' in this format:
Mon Sep 26
0
 

Author Comment

by:new_perl_user
ID: 36893336
oh my bad.  Yesterday in the sense I eas telling that it should get previous day data, but not grep for yestarday. It has to grep only date for previous day.

For example if I am executing the command tomorrow  it has to read the file and get today's data  as output.
0
 
LVL 38

Accepted Solution

by:
Gerwin Jansen, EE MVE earned 500 total points
ID: 36893444
The grep command is using a date command that gets you yesterday's Day Month Date.

You can try this separate command if you like:

date -d 'yesterday' +"%a %b %d"

Open in new window


If you run this today (Fri Sep 30), it will give you Thu Sep 29 as output. Just try the above command.
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question