Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Bash shell script - Cut all text before processing rest of text file.

Posted on 2006-03-27
17
Medium Priority
?
4,763 Views
Last Modified: 2008-01-09
I have multiple text files im processing with a script, the only problem is, each text file has a bunch of junk in the beginning of it that i don't need. The good thing, I think, is with each file there is a catch word named DIFF that is right before the important part of the text files.
This is usually at the end of a line after all the junk like the following.

TEST QUOTE DIFF

234234 23 23 asdf 1234 asdf     (IMPORTANT DATA STARTS HERE)

I'm taking this info after the DIFF line and processing it to a comma delimited file, Can I just hunt for DIFF and start processing from there? How would I start this in a bash script.

Thanks in advance!
0
Comment
Question by:cybrthug
  • 9
  • 7
17 Comments
 
LVL 85

Expert Comment

by:ozo
ID: 16304222
awk 'd{print}/DIFF/{d=1}' file | process
0
 

Author Comment

by:cybrthug
ID: 16304673
Ugh sorry I forgot something, there was more then one DIFF entry in the beginning, I'll up the points here for my mistake,

Instead of searching for DIFF I need to find:  EXTENSION DIFF,  there is a space in between so I need to start processing after I find the phrase "EXTENSION DIFF"
0
 
LVL 85

Accepted Solution

by:
ozo earned 400 total points
ID: 16304803
awk 'd{print}/EXTENSION DIFF/{d=1}' file | process
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 

Author Comment

by:cybrthug
ID: 16304829
Is there a way I can use this to process multiple files of the same type from the command line, maybe take output of your awk into a new file:

awk 'd{print}/QUANTITY/{d=1}'  323424.txt

take what i get from this file and save it as:   a323424.txt  
then move on to the next file in the directory and do the same, basically batch process the entire directory of text files.
0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 300 total points
ID: 16305070

#!/bin/sh
for file in *.txt
do
  awk 'd{print}/QUANTITY/{d=1}' $file >a$file
done
0
 

Author Comment

by:cybrthug
ID: 16305476
There seems to be 2 empty lines after the QUANTITY line ive pulled information after, Is there a way to remove the 2 blank lines also after the QUANTITY line? Thanks for all your help guys!
0
 
LVL 85

Expert Comment

by:ozo
ID: 16305589
Do you want to remove all blank lines?
The next 2 lines after QUANTITY?
Blank lines immediately following QUANTITY?
Is a line containing tabs and spaces blank?
0
 

Author Comment

by:cybrthug
ID: 16306341
After i cut all the text above the line that quantity is containted in using your awk statement, there are 2 blank lines following immediately below it, then the text that i need to process on the 3rd line. Is there a way to skip a blank line and move down to the next or skip 2 or more blank lines?
0
 
LVL 85

Expert Comment

by:ozo
ID: 16306477
We can skip any 2 immediately following lines, any number of immediately following blank lines, up to 2 immediately following blank lines, all blank lines,
skipping all blank lines may be the simplest:
awk '/./&&d{print}/QUANTITY/{d=1}' $file >a$file
although that does not consider a line containing spaces and tabs to be empty
0
 

Author Comment

by:cybrthug
ID: 16306577
It must have tabs in it being formatted the way it is, since that is still leaving them in. probably need to skip the following 2 blank spaced lines.
0
 
LVL 85

Expert Comment

by:ozo
ID: 16306650
awk 'NF&&d{print}/QUANTITY/{d=1}' $file >a$file
will not print any lines contining only spaces and tabs
awk 'NF&&d{d++}d>1{print}/QUANTITY/{d=1}'
will skip blank lines immediately following QUANTITY but print blank lines following a non-blank line following QUANTITY
0
 

Author Comment

by:cybrthug
ID: 16306716
They are removing everything including the QUANTITY line perfectly, but both those options still wont remove the next two lines. I did a check in the original text file and it appears each of the two lines appears completely blank as in you cant tab or move over with an arrow key, it immediately drops you down.
0
 
LVL 85

Expert Comment

by:ozo
ID: 16306773
can you do a cat -vet of the lines following QUANTITY
0
 

Author Comment

by:cybrthug
ID: 16306798
I get ^M$ only on those lines
0
 
LVL 85

Expert Comment

by:ozo
ID: 16306855
if there is a ^M at the end of all lines, then how about
 awk '/../&&d{print}/QUANTITY/{d=1}' $file >a$file
This will also skip lines that contain a single non ^M chararacter
0
 

Author Comment

by:cybrthug
ID: 16306874
Works great thank you very much, you have been very helpfull!
0
 

Author Comment

by:cybrthug
ID: 16316640
For some reason its stripping out something im needing to process the lines, when I run the awk above it parses the information I need fine but strips the $ which im needing for my processing script to work. Is there a way to leave in the $ without the ^M.  I will open another question and toss you some points after this since you've been of so much help. Thank you!
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses
Course of the Month14 days, 17 hours left to enroll

578 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question