Solved

Cat, Grep, Variable and a list file

Posted on 2006-11-03
9
3,175 Views
Last Modified: 2013-12-26
Ok here's the situation. I need to strip all lines from a long file which contain certain words. This list of words is ever-growing, and will probably stay that way. Currently I have a bash script that cats a file, then uses grep -v to strip that line, then it outputs it to another file after all the stripping is complete. At first it wasn't difficult, until I realised that I could only use so many instances of grep in a row before the script would break. So I was resolved to output it to another file and cat it again. I know there is an easier way to do this, I just don't know enough about shell scripting to figure it out.

Basicly what I want to have happen is I would like to maintain a separate file containing a list of words who's lines need to be stripped. I would then like the bash script to cat that file, grep for the variable and then output it to a new file with the words stripped.

Kinda like this.
cat filename.txt | grep -v $stripme > stripped.txt

I just don't know how to tell the script that $stripme is the contents of the stripme.txt file. The contents of the stripme.txt file can consist of 1 or more words, each on a new line. Example:
Item A
Bob
Snack Master Supreme
John


500 points for working code, and my thanks.
0
Comment
Question by:DarkHound
9 Comments
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866805

taking your sample above as "search.lst":
(assuming the file to process is "process.file")

#!/bin/sh

rm temp.file
cp process.file temp.file

cat search.lst | while read line; do
    grep -v "$line" temp.file > temp.file2
    mv temp.file2 temp.file
done


Cheers,
C.
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866843
however...

if your system has a decent implementation of "grep" the same can be done using the one-liner:

grep -vf stripme.txt filename.txt

this is probably the way to go :-)

Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17867296
I've tried both of these, neither seem to strip the lines.
0
Does Powershell have you tied up in knots?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

 
LVL 11

Expert Comment

by:cjjclifford
ID: 17867353
Hi,

Here's a test session showing both working as expected (for me on OSX)

$ cat > stripme.txt << EOF
> hello world
> this is a test
> goodbye creul world
> EOF
$
$
$
$ cat > filename.txt << EOF
> line should not be stripped
> hello world
> yet another line that should not be stripped
> this is a test line that should be stripped
> goodbye creul world, see you tomorrow
> last line should not be stripped
> EOF
$
$
$ cp filename.txt filename2.txt
$
$
$ cat stripme.txt | while read line; do
> grep -v "$line" filename.txt > filename.txt.tmp
> mv filename.txt.tmp filename.txt
> done
$
$ cat filename.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$
$
$ grep -vf stripme.txt filename2.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$


The "grep -vf" relies on specific Grep implementation, the shell approach should work on any grep implementation, as its not using anything special...

Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17867607
Ok well here's what my code looks like. It seems the same except for the mv bit at the end. The system seems to be processing the file, but the resulting file has exactly the same number of lines as the orginal file.

#!/bin/bash
rm temp.file
cp cigna temp.file
 
cat list.lst | while read line; do
   grep -v "$line" temp.file > cigtst.txt
done
echo "Rows successfully deleted"
0
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 17869323
grep -f stripme.txt -v filename.txt > stripped.txt
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17873406
you need to output to a temporary file, and make that the working file each iteration through the loop, otherwise only the last entry in "stripme.txt" will get eliminated.

But, as I suggested, and Ozo has also suggested, try the one-liner grep command, it may work immediately, much cleaner, and faster too,...


Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17880756
It seems that grep -vf stripme.txt filename.txt did not work, but Ozo's variation (grep -f stripme.txt -v filename.txt > stripped.txt) did. I'm not sure why there is a difference, probably due to the older version of AIX on this box.
0
 

Expert Comment

by:adamrosshill
ID: 22225017
I would love to know why this actually works..  if you would be so kind to as explain it.
0

Featured Post

Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Get filename and folder into excel 7 76
matchUp  challenge 6 65
post4 challenge 28 102
Not needed 13 113
This is to be the first in a series of articles demonstrating the development of a complete windows based application using the MFC classes.  I’ll try to keep each article focused on one (or a couple) of the tasks that one may meet.   Introductio…
Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
This video shows how to use Hyena, from SystemTools Software, to bulk import 100 user accounts from an external text file. View in 1080p for best video quality.

832 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question