Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Cat, Grep, Variable and a list file

Posted on 2006-11-03
9
Medium Priority
?
3,216 Views
Last Modified: 2013-12-26
Ok here's the situation. I need to strip all lines from a long file which contain certain words. This list of words is ever-growing, and will probably stay that way. Currently I have a bash script that cats a file, then uses grep -v to strip that line, then it outputs it to another file after all the stripping is complete. At first it wasn't difficult, until I realised that I could only use so many instances of grep in a row before the script would break. So I was resolved to output it to another file and cat it again. I know there is an easier way to do this, I just don't know enough about shell scripting to figure it out.

Basicly what I want to have happen is I would like to maintain a separate file containing a list of words who's lines need to be stripped. I would then like the bash script to cat that file, grep for the variable and then output it to a new file with the words stripped.

Kinda like this.
cat filename.txt | grep -v $stripme > stripped.txt

I just don't know how to tell the script that $stripme is the contents of the stripme.txt file. The contents of the stripme.txt file can consist of 1 or more words, each on a new line. Example:
Item A
Bob
Snack Master Supreme
John


500 points for working code, and my thanks.
0
Comment
Question by:DarkHound
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
9 Comments
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866805

taking your sample above as "search.lst":
(assuming the file to process is "process.file")

#!/bin/sh

rm temp.file
cp process.file temp.file

cat search.lst | while read line; do
    grep -v "$line" temp.file > temp.file2
    mv temp.file2 temp.file
done


Cheers,
C.
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17866843
however...

if your system has a decent implementation of "grep" the same can be done using the one-liner:

grep -vf stripme.txt filename.txt

this is probably the way to go :-)

Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17867296
I've tried both of these, neither seem to strip the lines.
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 11

Expert Comment

by:cjjclifford
ID: 17867353
Hi,

Here's a test session showing both working as expected (for me on OSX)

$ cat > stripme.txt << EOF
> hello world
> this is a test
> goodbye creul world
> EOF
$
$
$
$ cat > filename.txt << EOF
> line should not be stripped
> hello world
> yet another line that should not be stripped
> this is a test line that should be stripped
> goodbye creul world, see you tomorrow
> last line should not be stripped
> EOF
$
$
$ cp filename.txt filename2.txt
$
$
$ cat stripme.txt | while read line; do
> grep -v "$line" filename.txt > filename.txt.tmp
> mv filename.txt.tmp filename.txt
> done
$
$ cat filename.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$
$
$ grep -vf stripme.txt filename2.txt
line should not be stripped
yet another line that should not be stripped
last line should not be stripped
$


The "grep -vf" relies on specific Grep implementation, the shell approach should work on any grep implementation, as its not using anything special...

Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17867607
Ok well here's what my code looks like. It seems the same except for the mv bit at the end. The system seems to be processing the file, but the resulting file has exactly the same number of lines as the orginal file.

#!/bin/bash
rm temp.file
cp cigna temp.file
 
cat list.lst | while read line; do
   grep -v "$line" temp.file > cigtst.txt
done
echo "Rows successfully deleted"
0
 
LVL 84

Accepted Solution

by:
ozo earned 2000 total points
ID: 17869323
grep -f stripme.txt -v filename.txt > stripped.txt
0
 
LVL 11

Expert Comment

by:cjjclifford
ID: 17873406
you need to output to a temporary file, and make that the working file each iteration through the loop, otherwise only the last entry in "stripme.txt" will get eliminated.

But, as I suggested, and Ozo has also suggested, try the one-liner grep command, it may work immediately, much cleaner, and faster too,...


Cheers,
C.
0
 
LVL 1

Author Comment

by:DarkHound
ID: 17880756
It seems that grep -vf stripme.txt filename.txt did not work, but Ozo's variation (grep -f stripme.txt -v filename.txt > stripped.txt) did. I'm not sure why there is a difference, probably due to the older version of AIX on this box.
0
 

Expert Comment

by:adamrosshill
ID: 22225017
I would love to know why this actually works..  if you would be so kind to as explain it.
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Here is how to use MFC's automatic Radio Button handling in your dialog boxes and forms.  Beginner programmers usually start with a OnClick handler for each radio button and that's just not the right way to go.  MFC has a very cool system for handli…
Have you tried to learn about Unicode, UTF-8, and multibyte text encoding and all the articles are just too "academic" or too technical? This article aims to make the whole topic easy for just about anyone to understand.
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
Want to learn how to record your desktop screen without having to use an outside camera. Click on this video and learn how to use the cool google extension called "Screencastify"! Step 1: Open a new google tab Step 2: Go to the left hand upper corn…

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question