Solved

Cat, Grep, Variable and a list file

Posted on 2006-11-03
9
3,167 Views
Last Modified: 2013-12-26
Ok here's the situation. I need to strip all lines from a long file which contain certain words. This list of words is ever-growing, and will probably stay that way. Currently I have a bash script that cats a file, then uses grep -v to strip that line, then it outputs it to another file after all the stripping is complete. At first it wasn't difficult, until I realised that I could only use so many instances of grep in a row before the script would break. So I was resolved to output it to another file and cat it again. I know there is an easier way to do this, I just don't know enough about shell scripting to figure it out.

Basicly what I want to have happen is I would like to maintain a separate file containing a list of words who's lines need to be stripped. I would then like the bash script to cat that file, grep for the variable and then output it to a new file with the words stripped.

Kinda like this.
cat filename.txt | grep -v $stripme > stripped.txt

I just don't know how to tell the script that $stripme is the contents of the stripme.txt file. The contents of the stripme.txt file can consist of 1 or more words, each on a new line. Example:
Item A
Bob
Snack Master Supreme
John


500 points for working code, and my thanks.
0
Comment
Question by:DarkHound
9 Comments
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866805

taking your sample above as "search.lst":
(assuming the file to process is "process.file")

#!/bin/sh

rm temp.file
cp process.file temp.file

cat search.lst | while read line; do
    grep -v "$line" temp.file > temp.file2
    mv temp.file2 temp.file
done


Cheers,
C.
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866843
however...

if your system has a decent implementation of "grep" the same can be done using the one-liner:

grep -vf stripme.txt filename.txt

this is probably the way to go :-)

Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17867296
I've tried both of these, neither seem to strip the lines.
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17867353
Hi,

Here's a test session showing both working as expected (for me on OSX)

$ cat > stripme.txt << EOF
> hello world
> this is a test
> goodbye creul world
> EOF
$
$
$
$ cat > filename.txt << EOF
> line should not be stripped
> hello world
> yet another line that should not be stripped
> this is a test line that should be stripped
> goodbye creul world, see you tomorrow
> last line should not be stripped
> EOF
$
$
$ cp filename.txt filename2.txt
$
$
$ cat stripme.txt | while read line; do
> grep -v "$line" filename.txt > filename.txt.tmp
> mv filename.txt.tmp filename.txt
> done
$
$ cat filename.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$
$
$ grep -vf stripme.txt filename2.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$


The "grep -vf" relies on specific Grep implementation, the shell approach should work on any grep implementation, as its not using anything special...

Cheers,
C.
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 1

Author Comment

by:DarkHound
ID: 17867607
Ok well here's what my code looks like. It seems the same except for the mv bit at the end. The system seems to be processing the file, but the resulting file has exactly the same number of lines as the orginal file.

#!/bin/bash
rm temp.file
cp cigna temp.file
 
cat list.lst | while read line; do
   grep -v "$line" temp.file > cigtst.txt
done
echo "Rows successfully deleted"
0
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 17869323
grep -f stripme.txt -v filename.txt > stripped.txt
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17873406
you need to output to a temporary file, and make that the working file each iteration through the loop, otherwise only the last entry in "stripme.txt" will get eliminated.

But, as I suggested, and Ozo has also suggested, try the one-liner grep command, it may work immediately, much cleaner, and faster too,...


Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17880756
It seems that grep -vf stripme.txt filename.txt did not work, but Ozo's variation (grep -f stripme.txt -v filename.txt > stripped.txt) did. I'm not sure why there is a difference, probably due to the older version of AIX on this box.
0
 

Expert Comment

by:adamrosshill
ID: 22225017
I would love to know why this actually works..  if you would be so kind to as explain it.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
MFC Dialog 9 48
Unix / Linux grid computing 5 144
Prevent this page from creating additional dialogs. 3 390
post4 challenge 28 92
Introduction: Load and Save to file, Document-View interaction inside the SDI. Continuing from the second article about sudoku.   Open the project in visual studio. From the class view select CSudokuDoc and double click to open the header …
If you use Adobe Reader X it is possible you can't open OLE PDF documents in the standard. The reason is the 'save box mode' in adobe reader X. Many people think the protected Mode of adobe reader x is only to stop the write access. But this fe…
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
I designed this idea while studying technology in the classroom.  This is a semester long project.  Students are asked to take photographs on a specific topic which they find meaningful, it can be a place or situation such as travel or homelessness.…

914 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now