Solved

Need to cut lines from top of text file

Posted on 2008-10-27
9
677 Views
Last Modified: 2013-12-27
HI all,

I need to cut all line above the the DASH line (----- -- -- -----)

I've been using the cut and tail, but undercertain circumstance the amount of line above the dash line changes.  So hard coding to cut the top 7 lines doesnt work at certain times.... and its causing me issues because I have a script  that outputs the data CSV format and loads into a mysql database.  Needless to say, its causing issues with the database when loading a bunch of dashs....

So, my question is there some way to tell the system to cut only the line above the dash line ( and cut he dash line as well. basiclly keep everything under the dash line)

bash-3.00# cat pcavanilla
Downloading xref file to /var/tmp/patchdiag.xref
Trying http://sunsolve.sun.com/patchdiag.xref (1/1)
Using /var/tmp/patchdiag.xref from Oct/24/08
Host: fxssun (SunOS 5.10/Generic_137112-08/i386/i86pc)
List: missing
 
Patch  IR   CR RSB Age Synopsis
------ -- - -- --- --- -------------------------------------------------------
119964 10 < 11 ---  11 SunOS 5.10_x86: Shared library patch for C++_x86
bash-3.00#
 
here is what i'm doing now to cut the lines out and export to csv format

pca -l | tail +7 | while read line

do

  echo `hostname`";"`date '+ %m/%d/%y'";"``echo $line | awk '{print $1";"$2";"$4";"$5";"$6";"$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22}'`> ./pca2db.`hostname`

 

done

Open in new window

0
Comment
Question by:jedblack
  • 3
  • 3
  • 2
  • +1
9 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 22814358
try

loc=`cat file | grep -n '^--' | awk -F: '{ print $1 }'`
len=`cat file | wc -l`
lines=`expr $len - $loc`
tail -$lines file > newfile
0
 
LVL 22

Expert Comment

by:blu
ID: 22815029
You could also do it this way:

perl -ne 'print if /-----------/../IMPOSSIBLESTRING/' pcavanilla > newfile
0
 
LVL 1

Author Comment

by:jedblack
ID: 22816021
omarfarid,

I tried that, and it seemed to work...but this is the script i have in place right now to convert the PCA output to csv format, I've tried intergrating your code into the script...but it wont work.  Any ideas on how to modify the current script to include your solution?


bash-3.00# cat pca2csv.sh

#!/bin/bash

 

pca -l | tail +7 | while read line

do

  echo `hostname`";"`date '+ %m/%d/%y'";"``echo $line | awk '{print $1";"$2";"$4";"$5";"$6";"$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22}'`> ./pca2db.`hostname`

 

done

Open in new window

0
 
LVL 40

Expert Comment

by:omarfarid
ID: 22816098
what is the output of your script and how do you want in include mine into this?
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 22

Expert Comment

by:blu
ID: 22816184
You can easily do this with mine:

#!/bin/bash
 
pca -l |perl -ne 'print if /-----------/../IMPOSSIBLESTRING/' | while read line
do
  echo `hostname`";"`date '+ %m/%d/%y'";"``echo $line | awk '{print $1";"$2";"$4";"$5";"$6";"$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22}'`> ./pca2db.`hostname`
 
done

But actually, it would be even better to just to it all in perl in the first place. What is the format of your input file? In your first mesage you show a download of the patchdiag list, but that doesn't really jive with what you show later.  What is the field separator on the lines you input? What do you what the output to look like? I see semicolons and commas in there.
0
 
LVL 1

Author Comment

by:jedblack
ID: 22817278
We use a script called PCA to patch our solaris machine.  However, there is no reporting function built into the script.  So, I engineered a method to migrate the output produced by the script and batch load into a MYSQL database.  Below is what the un-altered raw output looks like....

-bash-3.00$ ./pca -l
Downloading xref file to /var/tmp/patchdiag.xref
Trying http://sunsolve.sun.com/patchdiag.xref (1/1)
Using /var/tmp/patchdiag.xref from Oct/24/08
Host: pca (SunOS 5.10/Generic_118833-24/sparc/sun4u)
List: missing
 
Patch  IR   CR RSB Age Synopsis
------ -- - -- --- --- -------------------------------------------------------
119254 42 < 59 RS-   5 SunOS 5.10: Install and Patch Utilities Patch
118666 -- < 17 -S- 111 JavaSE 5.0: update 16 patch (equivalent to JDK 5.0u16)
118676 -- < 03 --- 682 SunOS 5.10: patch for Solaris make and sccs utilities
118683 -- < 02 --- 410 SunOS 5.10: Patch for assembler
118735 01 < 02 --- 158 SunOS 5.10: patch usr/sbin/rpc.nisd_resolv
118777 -- < 13 R-- 164 SunOS 5.10: Sun GigaSwift Ethernet 1.0 driver patch
118814 -- < 01 --- 999 SunOS 5.10: patch platform/sun4u/kernel/tod/sparcv9/todsg
118824 -- < 02 --- 157 SunOS 5.10: patch usr/bin/sparcv9/sort
119578 29 < 30 R-- 739 SunOS 5.10: FMA Patch
118833 24 < 36 RS- 637 SunOS 5.10: kernel patch
118842 -- < 01 --- 999 SunOS 5.10: dada patch
118884 -- < 01 --- 999 SunOS 5.10: atomic.h patch
118918 19 < 24 RS- 630 SunOS 5.10: Solaris Crypto Framework patch
118981 -- < 03 --- 927 SunOS 5.10: Sun Quad FastEthernet qfe driver
119042 10 < 11 R-- 145 SunOS 5.10: svccfg & svcprop patch
119059 31 < 45 RS-  87 X11 6.6.2: Xsun patch
119090 -- < 31 R--  69 SunOS 5.10: Sun iSCSI Device Driver and Utilities
119130 26 < 33 R-- 621 SunOS 5.10: Sun Fibre Channel Device Drivers



Then I run this to turn the output in a CSV format to loading into the database.  Below is the command and the data output after running agaist the command
NOTE* part of the script inserts the date and hostname in the first two columns of the output
-------------------------
bash-3.00# cat pca2csv.sh
#!/bin/bash
 
pca -l | tail +7 | while read line
do
  echo `hostname`";"`date '+ %m/%d/%y'";"``echo $line | awk '{print $1";"$2";"$4";"$5";"$6";"$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22}'`> ./pca2db.`hostname`
 
done

-------------------------


bash-3.00# cat pca2db.js
js; 10/22/08;118677;02;03;---;656;SunOS 5.10_x86: patch for make utilities
js; 10/22/08;118778;07;11;---;329;SunOS 5.10_x86: Sun GigaSwift 1.0
js; 10/22/08;118816;03;04;---;369;SunOS 5.10_x86: /usr/xpg4/bin/awk patch
js; 10/22/08;119091;22;26;---;278;SunOS 5.10_x86: Sun iSCSI Driver
js; 10/22/08;119281;08;15;---;273;CDE 1.6_x86: Runtime library for
js; 10/22/08;119398;04;05;---;617;SunOS 5.10_x86: Patch for America issues
js; 10/22/08;119471;07;09;---;509;SunOS 5.10_x86: Sun Enterprise Array utilitie
js; 10/22/08;119535;09;13;---;376;SunOS 5.10_x86: Flash Archive
js; 10/22/08;119964;07;08;---;714;SunOS 5.10_x86: Shared library for
js; 10/22/08;119975;05;07;---;546;SunOS 5.10_x86: fp plug-in cfgadm
js; 10/22/08;120347;05;08;---;484;SunOS 5.10_x86: Common Fibre HBA Host Bus Li
js; 10/22/08;120349;01;02;---;673;SunOS 5.10_x86: Fibre Channel Port
js; 10/22/08;120411;17;26;---;286;SunOS 5.10_x86: Internet/Intranet Input Framework
js; 10/22/08;120754;02;05;---;572;SunOS 5.10_x86: Microtasking libraries patch


So, now here is the problem, I've noticed that depending on certains situations the  amount of top lines above the DASH(---) line differ.  I have it hard coded into my script to remove the top 7 lines, but sometime that leaves or takes too many lines away.

That is why I asked my original question....

Omarfarid, your methods works, but i'm not sure how to implement that in my script.
Blu, your method works...but it still leaves the DASH line





0
 
LVL 6

Expert Comment

by:Talmash
ID: 22817386
egrep -B1 '^\-+$' file

tal
0
 
LVL 40

Accepted Solution

by:
omarfarid earned 500 total points
ID: 22818747


pca -l > file

loc=`cat file | grep -n '^--' | awk -F: '{ print $1 }'`
len=`cat file | wc -l`
lines=`expr $len - $loc`
tail -$lines file > newfile
cat newfile  | tail +7 | while read line
do
  echo `hostname`";"`date '+ %m/%d/%y'";"``echo $line | awk '{print $1";"$2";"$4";"$5";"$6";"$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22}'`> ./pca2db.`hostname`
 
done
0
 
LVL 22

Expert Comment

by:blu
ID: 22821128
Okay, how about this:

#!/bin/perl
$,=';';
chomp ($h = `hostname`);
chomp ($d = `date '+%m/%d/%y'`);
while (<>) {
        if (/-----/) {$flag++;next;}
        next unless $flag;
        ($f1,$f2,$f3,$f4,$f5,$f6,$f7) = split(/\s+/,$_,7);
        print $h,$d,$f1,$f2,$f4,$f5,$f6,$f7;
}


Note that this is 3 processes created in total, while the script above uses 8 process plus 3 per line of input.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now