• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3219
  • Last Modified:

Cat, Grep, Variable and a list file

Ok here's the situation. I need to strip all lines from a long file which contain certain words. This list of words is ever-growing, and will probably stay that way. Currently I have a bash script that cats a file, then uses grep -v to strip that line, then it outputs it to another file after all the stripping is complete. At first it wasn't difficult, until I realised that I could only use so many instances of grep in a row before the script would break. So I was resolved to output it to another file and cat it again. I know there is an easier way to do this, I just don't know enough about shell scripting to figure it out.

Basicly what I want to have happen is I would like to maintain a separate file containing a list of words who's lines need to be stripped. I would then like the bash script to cat that file, grep for the variable and then output it to a new file with the words stripped.

Kinda like this.
cat filename.txt | grep -v $stripme > stripped.txt

I just don't know how to tell the script that $stripme is the contents of the stripme.txt file. The contents of the stripme.txt file can consist of 1 or more words, each on a new line. Example:
Item A
Bob
Snack Master Supreme
John


500 points for working code, and my thanks.
0
DarkHound
Asked:
DarkHound
1 Solution
 
cjjcliffordCommented:

taking your sample above as "search.lst":
(assuming the file to process is "process.file")

#!/bin/sh

rm temp.file
cp process.file temp.file

cat search.lst | while read line; do
    grep -v "$line" temp.file > temp.file2
    mv temp.file2 temp.file
done


Cheers,
C.
0
 
cjjcliffordCommented:
however...

if your system has a decent implementation of "grep" the same can be done using the one-liner:

grep -vf stripme.txt filename.txt

this is probably the way to go :-)

Cheers,
C.
0
 
DarkHoundAuthor Commented:
I've tried both of these, neither seem to strip the lines.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
cjjcliffordCommented:
Hi,

Here's a test session showing both working as expected (for me on OSX)

$ cat > stripme.txt << EOF
> hello world
> this is a test
> goodbye creul world
> EOF
$
$
$
$ cat > filename.txt << EOF
> line should not be stripped
> hello world
> yet another line that should not be stripped
> this is a test line that should be stripped
> goodbye creul world, see you tomorrow
> last line should not be stripped
> EOF
$
$
$ cp filename.txt filename2.txt
$
$
$ cat stripme.txt | while read line; do
> grep -v "$line" filename.txt > filename.txt.tmp
> mv filename.txt.tmp filename.txt
> done
$
$ cat filename.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$
$
$ grep -vf stripme.txt filename2.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$


The "grep -vf" relies on specific Grep implementation, the shell approach should work on any grep implementation, as its not using anything special...

Cheers,
C.
0
 
DarkHoundAuthor Commented:
Ok well here's what my code looks like. It seems the same except for the mv bit at the end. The system seems to be processing the file, but the resulting file has exactly the same number of lines as the orginal file.

#!/bin/bash
rm temp.file
cp cigna temp.file
 
cat list.lst | while read line; do
   grep -v "$line" temp.file > cigtst.txt
done
echo "Rows successfully deleted"
0
 
ozoCommented:
grep -f stripme.txt -v filename.txt > stripped.txt
0
 
cjjcliffordCommented:
you need to output to a temporary file, and make that the working file each iteration through the loop, otherwise only the last entry in "stripme.txt" will get eliminated.

But, as I suggested, and Ozo has also suggested, try the one-liner grep command, it may work immediately, much cleaner, and faster too,...


Cheers,
C.
0
 
DarkHoundAuthor Commented:
It seems that grep -vf stripme.txt filename.txt did not work, but Ozo's variation (grep -f stripme.txt -v filename.txt > stripped.txt) did. I'm not sure why there is a difference, probably due to the older version of AIX on this box.
0
 
adamrosshillCommented:
I would love to know why this actually works..  if you would be so kind to as explain it.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now